Conjugate nuclei

I'm trying to find an explanation for the difference in binding energies between conjugate nuclei being governed by the Coulomb interaction. I was looking at the semi-empirical mass formula and trying to draw some conclusion.

We have that:

$$R = R_0 A^{1/3}$$

and:

$$Z = A-N$$

so for the Coulomb term I find that it gets a $$A^{5/3}$$ dependence and of all the terms in the formula that one dominates in terms of $$A$$ dependence but it's not entirely clear as to why does the Coulomb interaction govern the difference in binding energies for conjugate nuclei. How do we justify this?

$$E_B(N,Z)=a_VA-a_SA^{2/3}-a_C\frac{Z(Z-1)}{A^{1/3}}-a_A\frac{(N-Z)^2}{A}+\delta(N,Z)$$
the volume and surface terms only depend on $$A$$, so they give the same value for a nucleus and its mirror nucleus, since the mirror nucleus is obtained by changing $$N\leftrightarrow Z$$, so $$A$$ is fixed. The asymmetry term depends on $$(N-Z)^2/A$$, so it's also the same when $$N$$ and $$Z$$ are interchanged. The pairing term $$\delta(N,Z)$$ is either $$0$$ or only depends on $$A$$. So the only term that changes between a nucleus and its conjugate is the Coulomb term and the difference in their binding energies $$\Delta E_B=E_B(N,Z)-E_B(Z,N)$$ must come from this term.