Why is argon a noble gas but not, say, beryllium or palladium? Why is argon a noble gas given that the 3d subshell is still empty?
More generally, why is it that the filling of a p sub-shell makes an element noble rather than s, d, or f sub-shells, or completed n-levels?
Let's start with the prior that in a hydrogen atom the principle quantum number determines the energy - so maybe full shells correspond to noble gases.
Now, I understand that in helium the n=1 level is full, so it is a noble gas.
Then, beryllium has a full 2s subshell, but empty 2p, so it is a metal.
Neon has the n=2 level full, and is a noble gas.
Magnesium has a full 3s subshell, but not a full n=3 level and is a metal - so far so good.
But, oops, argon has full 3s and 3p subshells but empty 3d, yet it is a noble gas.
At this point we look at the aufbau principle and say "ah, your prior is no good, the 4s is lower energy than 3d, and this is the first time it has kicked in - so there you go".
But, as we continue down the periodic table there are more noble gasses and they happen only with the filling of a p subshell.  Why is krypton noble, but not zinc?  Even palladium cleanly fills n=4 up through 4d (with only 4f empty) but it is still not a noble gas.  Ytterbium fills an f subshell for the first time and it isn't noble.  What makes the filling of a p subshell so specially inert compared to s, d, and f subshells?
The aufbau principle does not explain this.  Draw the classic "diagonal aufbau chart" and circle where the noble gasses are.  It doesn't add any insight.  Noble gasses only happen when you fill a p subshell.  Why?
Edited to add:  When an electron is promoted out of a two electron s-shell into an empty p-shell the spin will flip and energy is saved because of the Pauli exchange (two same spin electrons can avoid each other).  The one remaining s electron, and the one new p electron benefit from each having a sub-shell to themselves - kinda a push vs. the s to p promotion.  This, to me, intuitively explains why full s shells  aren't noble.   
User4552 points out that brute force calculation shows that energy gaps after a p shell is filled are large.  But it doesn't help my intuition to just hear "the computer calculations reveal that ...".  So can anyone help with intuition on why full p-shells, rather that full d-shells, are noble?
 A: As a proxy for an atom's reluctance to form chemical bonds, we can use the energy required to lift one of its outer electrons to the next higher orbital, pretending as usual that the "orbital" concept is still at least approximately valid in a multi-electron atom. Then the question is why this energy gap tends to be significantly larger after filling a p-shell ($\ell=1$) than it is after filling s- or d- or f-shells ($\ell=0,2,3$).
As usual, let $n$ and $\ell$ denote the radial and angular quantum numbers, respectively. Let's accept that orbitals are filled in order of increasing $n+\ell$, and then in order of increasing $n$ whenever the first rule is neutral. (Reality is a little more complicated, but these rules mostly work pretty well.) The following list shows the filling order defined by these two rules, along with the number of nodes in the orbital wavefunction's radial factor:
\begin{align}
n+\ell &\hskip1cm (n,\ell) &\hskip1cm \text{radial nodes} & \\
\hline
1 &\hskip1cm (1,0) &\hskip1cm 0 &\hskip1cm  \text{helium}\\
2 &\hskip1cm (2,0) &\hskip1cm 1 \\
3 &\hskip1cm (2,1) &\hskip1cm 0 &\hskip1cm  \text{neon}\\
3 &\hskip1cm (3,0) &\hskip1cm 2 \\
4 &\hskip1cm (3,1) &\hskip1cm 1 &\hskip1cm  \text{argon} \\
4 &\hskip1cm (4,0) &\hskip1cm 3 \\
5 &\hskip1cm (3,2) &\hskip1cm 0 \\
5 &\hskip1cm (4,1) &\hskip1cm 2 &\hskip1cm  \text{krypton} \\
5 &\hskip1cm (5,0) &\hskip1cm 4 \\
6 &\hskip1cm (4,2) &\hskip1cm 1 \\
6 &\hskip1cm (5,1) &\hskip1cm 3 &\hskip1cm  \text{xenon} \\
6 &\hskip1cm (6,0) &\hskip1cm 5 \\
7 &\hskip1cm (4,3) &\hskip1cm 0 \\
7 &\hskip1cm (5,2) &\hskip1cm 2 \\
7 &\hskip1cm (6,1) &\hskip1cm 4 &\hskip1cm  \text{radon} \\
7 &\hskip1cm (7,0) &\hskip1cm 6 \\
8 &\hskip1cm (5,3) &\hskip1cm 1 \\
\end{align}
In every noble-gas case, and only in these cases, the next available level is an s-orbital ($\ell=0$), which has more radial nodes than the levels immediately before or after it in the sequence. (This pattern works for helium, too, even though helium doesn't have a filled p-shell.) If we could understand intuitively why energy gap between the ground state and the first excited state is relatively large when the first excited state is an s-orbital, then we would have at least a partial answer to the question. 
The order in which the shells are filled (the sequence shown above) indicates that radial nodes are more costly than "azimuthal" nodes, because for a given $n+\ell$, the cases with smaller $\ell$ (fewer "azimuthal" nodes) have higher energy. Accepting this trend as an axiom, we can focus our intuition on the radial part.
Intuitively, if we think of the radial nodes as "no-fly zones" for that electron, then having a larger number of radial nodes may correspond to having less freedom to re-arrange the multi-electron system to minimize the energy in the presence of electron-electron interactions. Here's the key idea: by analogy with a construction-induced traffic jam, the impact of each additional node (each additional no-fly zone) could be an increasing function of the number of nodes already present. This picture suggests that lifting an electron from a filled p-shell up to the next available s-shell should be more costly than, say, lifting an electron from a filled d-shell up to the next available p-shell, because the former lift requires increasing an already-larger number of radial nodes. Using the magnitude of the energy-gap as a proxy for an atom's reluctance to form bonds, this suggests that the noble gases should be less reactive (relatively), at least among atoms with the same value of $n+\ell$.
The key idea is that the energy-cost of each additional radial node is an increasing function of the number of radial nodes already present. This seems to be consistent with the information I've seen, but I have no real justification for anticipating it except for the dubious traffic-jam analogy. Even if the intuition is correct, it isn't quantitative enough to predict just how noble the noble gases are. It is suggestive at best, but at least it doesn't rely entirely on a computer. That's why I thought it was worth posting.
A: Aufbau arises mainly from the fact that the screened potential is not the same as the $1/r$ potential of hydrogen. In the screened potential, some levels are pushed up in energy and some are pushed down. Basically the high-l orbitals have less probability concentrated near the nucleus, where the screened potential is the most negative, so they're pushed up in energy relative to the low-l orbitals.
In addition to this, you get effects that are the way they are not because of any semiclassical argument because of the way the Schrodinger equation works. You're basically diagonalizing a big matrix, and there is no guarantee that the results will come out to be easy to explain.  Sometimes such results can be at least partially described by heuristics, such as a tendency for energy levels to "repel" each other, although it's not actually a physical repulsion. For this "level repulsion" to happen, the levels have to have the same angular momentum. (It's actually pretty surprising that this kind of level repulsion leaves the $n$ shells degenerate in hydrogen. You would think it would split them up.)
Noble gases exist when there is a large gap between the single-particle energy levels. These gaps do not occur for the screened potential at the same particle numbers where they would occur for hydrogen.
Here are some energy levels of states in noble gases, calculated using the Hartree-Fock method, from Johnson, Lectures on Atomic Physics, p. 86:

The Hartree-Fock method calculates the energies self-consistently for a particular atom, so the energies are different for the different atoms. It would be convenient to have the calculation for something like potassium, so we could see the gap between the last filled state in argon and the final electron for potassium, but he doesn't actually give those. However, you can clearly see the relevant shell gap in the calculation for krypton. There is a gap of 4.5 Ha between 3p and 3d. This is considerably larger than the gap between 3s and 3p (2.5 Ha) and between 3d and 4s (2.7 Ha).

So - what is special about p sub shells such that when they fill there is a larger gap up to the next level than when s,d, or f sub shells fill?

There is nothing special about p shells. These calculations are based on diagonalizing a big matrix. There is no particular reason to expect the spacing between subshells to follow any simple rules.

Beryllium isn’t a noble gas.

If you look at the levels for argon, you can see that the gap between 2s and 2p is less than 3 Ha, while the gap between 2p and 3s is about 8 Ha. The reason for the sizes of the gaps is ultimately just that that's what you get when you diagonalize the matrix. However, I don't find the result particularly surprising, since the unscreened potential has 2s degenerate with 2p, and a big gap between 2p and 3s.
A: In my opinion @user4552's answer captures the key points from a theoretical perspective: 


*

*The reactivity of an atom is determined by it's energy level structure. 

*A simple semi-classical model (like Bohr's model) is insufficient to capture the full energy level structure, due to screening effects and interactions -- i.e. spring orbit coupling. The Hartree Fock method yields satisfactory results.


Nevertheless, I feel like we should point out that the name "nobel gas" was  used before we understood the physics. Thus, the distinction becomes somewhat arbitrary, if the electron number becomes "large". In order to emphasise this argument,  I took the ionisation energies (from NIST database) and plotted them as a function of the number of electrons (=#protons):
 
By interpreting this graph see that


*

*the electrons of nobel gases are "strongly" bound to their nucleus (="large" ionisation energy). Hence, nobel gases "do not like to share" one of their electrons. 

*nobel gases "do not like to accommodate" an additional electron. This can be seen by considering the atom which has one additional proton. These (alkali) atoms "like to get rid" of their additional electrons -- alkali atoms are "easy" to ionise. Thus, by adding an additional proton to a nobel gas atom does not increase the ionisation energy. 


Combining these two observations, the ionisation energy is a simple measure to understand nobel gases: Neither do accommodate an additional electron, nor do they share an electron with other atoms. Thus, they are "reluctant" to react with other atoms.  
In the graph I mark the nobel gases in red. We see that the ionisation energy increases steadily towards the nobel gases, it reaches it's local maximum at the nobel gas atom, and then takes an  abrupt, negative step.
However, this structure is also visible for the atoms with filled $d$-shell  (marked in green): Here we also observe a steady increase in the ionisation energy, a local maximum followed by an abrupt, negative step. Thus, the distinction between nobel gases and atoms with filled $d$ shells is not significant.
Furthermore, with increasing electron number both (a) the ionisation energy and (b) the step size in energy become approx. equal. Hence, 
calling radon a nobel gas and mercury a transition metal is a matter of convenience (consistency with the historical system/nomenclature) and not a statement of "stability" (in my opinion). 
Using intuitive, semi-classical arguments to explain quantum effects is a sure path towards failure. Nevertheless, since Slater has already done this, there is little harm to restate the idea. Slater modelled the  screening of the electrons by 
$$
E = - h c R_H \left(\frac{Z-S}{n^\star}\right)^2
$$
where $R_H$ is Rydberg's constant, $S$ is a screening factor and $n^\star$ is an effective quantum number. Now, the screening factor depends on the number of electrons within a single shell. (simplified)


*

*Helium: Before ionisation there are two $s$-shell electrons. Thus the screening factor is $1/2$. After the ionisation the screening factor is zero. The difference of these screening factors determines how strong the electrons are bound.

*Lithium: Before ionisation there are three $s$-shell electrons. This yields a screening factor $3/2$. After the ionisation the screening factor is $1/2$. 


Using this kind of argument, we see that the $d$-shell electrons are strongly screened. Because a filled $d$-shell contains ten electrons, the screening difference between ten and nine electrons is "rather small". In contrast, the filled $p$-shell contains only six electrons. Thus, the difference in screening effect is larger as in the $d$-shell. 
Using Slater's argument, one could conclude that the filled $s$-shells should form the nobel gases. So it's only an argument against the $d$ and $f$ shells.
A: The definition of a noble gas has changed over the years as our understanding of the atom improved. The first relevent definition is that a noble gas is one that has a completed octet. In other words the outer most shell is supposed to have $8$ electrons (with the obvious exception of helium where the last shell can have only $2$ electrons). This definition was found purely experimentally. Later on as quantum mechanics developed this definition changed. Now the definition is that a noble gas has a completed $p$-orbital (with the exception of helium again). Hence one answer to your question is that it is defined that way. But this answer is really unsatisfactory. 
For a satisfactory answer one must ask what makes a noble gas noble? As we all know noble gases are not supposed to react, i.e. they should not accept, donate or share electrons easily. This means that the electron affinity and ionisation energy of noble gases should be very high. I am not going to mention the exact numbers but they can be checked here. The electron affinity and ionisation energy values are higher for a filled $p$-orbital than a filled $d$-orbital. This can be shown by a simple example of Argon and Zinc. To add an electron to Zinc is easier than Argon because in Zinc the screening will be provided significantly only by the $4s$ orbital. But in argon the screening will be provided by the $3p$ orbital, which is significantly higher. Hence Argon will be more stable than Zinc. Zinc is not a noble element since it has very different properties like lower screening effect, EA, IE, etc when compared to other noble gases. But this does not mean that it does not form bonds. Only after Quantum Mechanics we were able to explain the existence of compounds like $XeF_4$. These compounds were a major drawback of the previous theory. 
Hope this helps.
