Suppose we have two particles that attract each other. This could be an electron attracted to the nucleus by the electromagnetic force, or two nucleons attracted by the strong force, but let's keep it general for now. Suppose these two particles are separated by a distance $r$:
If the particles were for example an electron and a proton there would be an attraction between them and as we decrease $r$ the energy $E$ will decrease according to:
$$ E = -\frac{A}{r} $$
for some constant $A$ that tells us how strong the attractive force between the particles is. The particles want to reduce their energy, so they will try and make the distance $r$ between them as small as possible i.e. they will try to merge together.
But Heisenberg's uncertainty principle tells us that:
$$ \Delta x \Delta p \ge \frac{\hbar}{2} $$
that is, if we localise a particle to within a distance $\Delta x$ its momentum becomes uncertain by an amount $\Delta p$. In our system of two particles we can, in an arm waving way, say that the uncertainty in position is around the distance between the particles. A quick rearrangement of the equation above tells us that the momentum uncertainty is related to $r$ by:
$$ \Delta p \ge \frac{\hbar}{2r} $$
The reason this matters is that the energy of a system is related to its momentum by:
$$ E = \frac{p^2}{2m} $$
and if we take the $\Delta p$ we calculated above and put it in this equation we get:
$$ E = \frac{\hbar^2}{8r^2m} $$
So as we move the particles together the uncertainty principle means their energy increases, and this opposes the attractive force between the particles. The particles are going to end up at a distance where these two effects balance out, i.e.
$$ \frac{A}{r} = \frac{\hbar^2}{8r^2m} $$
and rearranging this for $r$ gives:
$$ r = \frac{\hbar^2}{8Am} $$
Since this is a very approximate argument let's ignore the constants and just write:
$$ r \propto \frac{1}{Am} $$
And this immediately tells us why nuclei are smaller than atoms. The mass of nucleons is about 2,000 times greater than the mass of electrons, and our equation tells us that size reduces as mass increases. Also nuclear forces are stronger than electromagnetic forces, i.e. the constant $A$ is greater for nuclear forces, and the equation tells us that as $A$ increases $r$ decreases. Both effects mean the size of nuclei is going to be smaller than the size of atoms.