As far as I know, it is the fermi repulsion that gives a collection of protons or neutrons its finite size. But this only acts on indistinguishable fermions. If the protons and neutrons do not repel each other by any other means, there would be no reason why adding neutrons to a nucleus would increase the volume of the collection of the protons.
What I'm asking, I think, is how protons and neutrons repel each other in the nucleus. Is it because of a repulsive term in the strong/weak force? Or is it fermi repulsion resulting from neutrons and protons continuously changing into each other through $W^{+/-}$ and $\pi^{+/-}$ exchange?