Why are do neutral atoms shrink as their valence shells approach 8 electrons?

Why do neutral, unbonded atoms shrink in size as they approach having 8 electrons in their valence shells? A good example is elements 3 through 10 in this table, that is, lithium (1 valence electron) through neon (8 valence electrons).

For posterity, below is my earlier incorrect version of my question. It's incorrect because it based on a table that used the atomic radii of bonded atoms to describe their sizes. That is a very different measurement from the sizes of neutral atoms, since for bonded atoms you are tossing in additional electrons via the bonds. Very small atomic radii for halogens are easy to explain in that case, since the incomplete shells must fight very hard over the shared electrons to complete their own octets, making their bonding radii anomalously short. This question had an interesting back-and-forth, and I thank the two contributors both who both caught my chart interpretation error and answered the real and interesting question that lurked beneath my incorrect one.

If you look here at the relative sizes of charge-neutral atoms, you can see something I never noticed or even thought about until I tried to think through this question in the new Chemistry beta.

Why are nearly-complete atomic shells, such as the seven valence electrons of fluorine, so compact in comparison to the bloated-but-very-stable situation created by adding one more electron to make neon?

I thought I could answer that in terms of Pauli exclusion and wound up completely baffled. Can anyone help on this one? Why does "almost complete" equal "very small" in a charge-neutral atom?

-
I'm not sure I understand all of the relevant factors in computing atomic radii, but this table of theoretical values seems to contradict yours. –  Phil Reinhold Apr 27 '12 at 4:43
Phil, look along a row e.g. Boron to Neon, Scandium to Zin or lanthanum to Lutetium and you'll see the shrinkage Terry is referring to. –  John Rennie Apr 27 '12 at 6:59
There would seem to be a problem here: Terrys' figures show noble gases with larger atomic radii than other elements on the same row; Phil's chart shows noble gases having smaller atomic radii than other elements on the same row. Terry's question rests on the assumption that the noble gases are larger. @John's answer rests on the assumption that they are smaller. –  EnergyNumbers Apr 27 '12 at 8:13
Oops yes. I think the radii in Terry's link come from bond lengths. The noble gases are so inert that the bond lengths are large. Presumably for Neon the bond length is the interatomic spacing in the solid, since it doesn't form any compounds. The radii in Phil's link come from a calculation of the electron density of an isolated atom. –  John Rennie Apr 27 '12 at 8:29
@PhilReinhold, all, thanks! Do you mean this chart? I wondered right after posting how that site was defining atomic size. That could be a tough experimental measurement for atomic fluorine, so the site I used must have depended on bond lengths... which of course would completely hoses the premise of my question by adding in more electrons! My assumption was that an unexcited fluorine atom floating in empty space that is probed -- hmm, how, by very hard X-rays? -- would show a much more compact profile than does neon. –  Terry Bollinger Apr 27 '12 at 20:13

I'm not sure how easy it would be to be to give a rigorous explanation of this. Here's an explanation based on the atomic orbital approach to the electronic structure of atoms. This is only an approximation, but I think it gives a good flavour of what is going on.

To clarify Terry's question: if you take the atomic orbital approximation to electronic structure you can ask what happens when e.g. the $p$, $d$, or $f$ orbitals are filling up. For example, along the first row of the periodic table from Boron to Neon the $2p$ orbital is filling up, or along the first row of the transition metals from Scandium to Zinc the $3d$ orbital is filling up, and in both cases the radius of the atom decreases.

The $p$, $d$ and $f$ (and higher if we ever make atoms that heavy) orbitals are generally expressed as a product of a radial and angular part, and for this question it's the radial part we have to consider. The atoms are shrinking because the radial part shrinks towards the nucleus, and as you've probably guessed it's because of the increasing nuclear charge attracting the electrons.

Consider the first row from Boron to Neon, i.e. the $2p$ shell is filling. You can get six electrons into this shell. They all share the same radial part of the atomic orbital and they're distributed between the three angular orbitals, $2p_x$, $2p_y$ and $2p_z$. As you add more electrons to the $2p$ orbital (and increase the nuclear charge) you get some mutual screening of the nuclear charge. However the electrons avoid each other as much as possible, so although they do screen each other from the increased nuclear charge the increase in screening is less than the increase in the nuclear charge. The result is that the effective potential felt by the electrons deepens and the radial part of the $2p$ orbital shrinks towards the nucleus. That's why Neon is the smallest atom in this row. When you add another electron it has to go into a new atomic orbital, the $3s$, and the radial part of the $3s$ extends further than the radial part of the $2p$ so the atom suddenly grows.

The reason all this is an approximation is because the splitting of the electron configuration into separate orbitals only works if there are no electron correlation terms i.e. if all the electrons feel a central averaged field, and of course this isn't true. However it's a pretty good approximation for light atoms and I think gives a reasonable explanation of Terry's question.

-
John Rennie, since the time stamp on your answer is earlier than the "oops" on your comment (my oops inherited by you, to be precise about it), should I assume the above answer is OBE? My thanks in any case, as you've done an interesting and detailed analysis here. I'll read it more carefully now that I've gotten the OBE issue duly noted. And, um... how do I score this? You and @PhilReinhold have both been been very helpful, but I now suspect that your answer above may be... well... a really good explanation for a phenomenon that does not actually exist? :) –  Terry Bollinger Apr 27 '12 at 20:27