Electron shells in atoms: What causes them to exist as they do? I have seen similar posts, but I haven't seen what seems to be a clear and direct answer.
Why do only a certain number of electrons occupy each shell? Why are the shells arranged in certain distances from the nucleus? Why don't electrons just collapse into the nucleus or fly away?
It seems there are lots of equations and theories that describe HOW electrons behave (pauli exclusion principle), predictions about WHERE they may be located (Schrödinger equation, uncertainty principle), etc. But hard to find the WHY and/or causality behind these descriptive properties.  What is it about the nucleus and the electrons that causes them to attract/repel in the form of these shells at regular intervals and numbers of electrons per shell?
Please be patient with me, new to this forum and just an amateur fan of physics.
 A: First of all, strictly speaking, electron shells (as well as atomic orbitals) do not exist in atoms with more than one electron. Such physical model of an atom is simplified (and often oversimplified), it arises from a mathematical approximation, which physically corresponds to the situation when electrons do not instantaneously interact with each other, but rather each and every electron interacts with the average, or mean, electric field created by all other electrons.
This approximation is known as mean field approximation, and the state (or, speaking classically, the motion) of each and every electron in this approximation is independent of the state (motion) of all other electrons in the system.
Thus, the physical model which arises due to this approximation is simplified, and, not surprisingly, it is often referred to as independent electrons model.
So, the question why nature works in this way, does not make a lot of sense, since nature actually does not work this way. Except for systems with only one electron, like, for instance, hydrogen atom. In any case the answer to the question why something works in this or that way in physics is pretty simple: according to the laws of a particular physical theory, say, quantum mechanics. And I could not explain to you quantum mechanics here in just a few sentences. You need to read some books.
But if your question is why nature works in this way according to quantum mechanics, i.e. why things in quantum mechanics are the way they are, then I would like to quote Paul Dirac:

[...] the main object of physical science is not the provision of
  pictures, but is the formulation of laws governing phenomena and the
  application of these laws to the discovery of new phenomena. If a
  picture exists, so much the better; but whether a picture exists or
  not is a matter of only secondary importance. In the case of atomic
  phenomena no picture can be expected to exist in the usual sense of
  the word 'picture', by which is meant a model functioning essentially
  on classical lines. One may, however, extend the meaning of the word
  'picture' to include any way of looking at the fundamental laws which
  makes their self-consistency obvious. With this extension, one may
  gradually acquire a picture of atomic phenomena by becoming familiar
  with the laws of the quantum theory. 

From "The Principles of Quantum Mechanics", §4.
A: A big part of it can be explained by combining the constraints of quantum mechanics with the geometry of angular momentum.
For the special case of the hydrogen atom, it turns out that when you solve the equations of motion for an electron near a proton, you can't give the electron any old energy. There's a set of energies that are allowed; all others are excluded. You can put these energies in order, starting from the most tightly bound, and give each one a number.  This is often called the "principal quantum number," $n$, and it can be any positive integer. The binding energy of an electron in the $n$-th state is $13.6\,\mathrm{eV}/n^2$.
You can also ask (again, using the mathematical tools of quantum mechanics) whether the electron can carry angular momentum. It turns out that it can, but again that the amount of angular momentum it can carry comes in lumps, and again we can put the angular momentum states in order, starting with the least. Unlike with the principal quantum number, it makes sense to talk about an atom whose angular momentum is zero, so the "angular momentum quantum number" $\ell$ starts counting from zero. For a very sneaky reason, $\ell$ must be smaller than $n$. So an electron in its ground state, $n=1$, must have $\ell=0$; an electron in the first excited state $n=2$ may have $\ell=0$ or $\ell=1$; and so on.
Now once you have started to ask about angular momentum you start to think about planets orbiting a star, and that suggests a question: what is the orientation of the orbit? Must all the electrons orbit in the same plane, like all the planets in the solar system are found roughly along the plane of the ecliptic? Or can electrons orbiting a nucleus occupy any random plane, the way that comets do? This is a question you can also address with quantum mechanics. It turns out (again) that only certain orientations are allowed, and the number of orientations that are allowed depends on $\ell$, and that you can put the orientations in order. For a state with $\ell=0$ there is only one orientation permitted. For a state with $\ell=1$ there are three orientations permitted; sometimes it makes sense to number them with the "angular momentum projection quantum number" $m \in \{-1,0,1\}$, and other times it makes sense to identify them with the three axes $x,y,z$ of a coordinate system. For $\ell=2$, likewise, it sometimes makes sense to identify orientations $m \in \{-2,-1,0,1,2\}$, and other time to identify the orientations with electrons along the axes and planes of the coordinate system. I think the chemists may even have a geometrical interpretation for the seven substates of $\ell=3$, but I'm not familiar with it.
When you start to add multiple electrons to one nucleus, several things change — most notably the interaction energy, since the electrons interact with each other as well as with the nucleus. The basic picture, that each electron must carry integer angular momentum $\ell$ which may lie on any of $2\ell+1$ directions, remains unchanged. But there is one final quirk: each state with a given $n,\ell,m$ may hold no more than two electrons! We can fit this into our picture by assigning each electron a fourth quantum number $s$, called the "spin quantum number" for reasons that you should totally look up later, which can only take two values. Now we have a very simple rule: a "state" described by the four numbers $n,\ell,m,s$ can hold zero or one electrons at a time.
After that preamble, have a look at a periodic table:



*

*Over on the left are two columns of highly reactive elements. These have the outermost electron with $\ell=0$ (one value of $m$ allowed, two values of $s$).

*Over on the right are six columns of (mostly) nonmetals. These have the outermost electron with $\ell=1$ (three values of $m$ allowed, times two values of $s$)

*In the middle are ten columns of metals. These have outermost electrons with $\ell=2$ (five values of $m$ allowed, times two values of $s$).

*Appended on the bottom of the chart, because there's too much blank space on the page if they're inserted between columns two and three, are fourteen columns of lanthanides and actinides. These have outermost electrons with $\ell=3$ (seven values of $m$, times two values of $s$).
This simple model doesn't explain everything about the periodic table and electron shells. My description puts helium in the wrong spot (it's not a reactive metal because the most tightly bound electron shell is special), and the heavier metals leak over into the $\ell=1$ block. You have to do some serious modeling to understand why the $\ell=2$ electrons aren't allowed until the fourth row, rather than the third row. Protons and neutrons in the nucleus have the same sort of shell structure, but nuclear magic numbers don't always occur after the filling of an $\ell=1$ shell the way the noble gases do. But that is about the shape of things.
A: Any answer based on analogies rather than mathematics is going to be misleading, so please bear this in mind when you read this.
Most of us will have discovered that if you tie one end of a rope to a wall and wave the other you can get standing waves on it like this:

Depending on how fast you wave the end of the rope you can get half a wave (A), one wave (B), one and a half waves (C), and so on. But you can't have 3/5 of a wave or 4.4328425 waves. You can only have a half integral number of waves. The number of waves is quantised.
This is basically why electron energies in an atom are quantised. You've probably heard that electrons behave as waves as well as particles. Well if you're trying to cram an electron into a confined space you'll only be able to do so if the electron wavelength fits neatly into the space. This is a lot more complicated than just waving a rope because an atom is a 3D object so you have 3D waves. However take for example the first three $s$ wavefunctions, which are spherically symmetric, and look how they vary with distance - you get (these are for a hydrogen atom) $^1$:

Unlike the rope the waves aren't all the same size and length because the potential around a hydrogen atom varies with distance, however you can see a general similarity with the first three modes of the rope.
And that's basically it. Energy increases with decreasing wavelength, so the "half wave" $1s$ level has a lower energy than the "one wave" $2s$ level, and the $2s$ has a lower energy than the "one and a half wave" $3s$ level.

$^1$ the graphs are actually the electron probability distribution $P(r) = \psi\psi^*4\pi r^2$. I did try plotting the wavefunction, but it was less visually effective.
A: John Rennie gave a nice answer based on the De Broglie hypothesis, however he didn't try the hard part: "Why do only a certain number of electrons occupy each shell?" so let me try!
In quantum mechanics particles are described by wave functions. All the observable properties of a particle (like its position) are related to the square of the wave function, so its sign does not really matter.
You can write a global wave function for a system of more particles. Let's consider two identical particles: the properties of the system should stay the same if they are swapped, this means that the global wave function must in principle:


*

*stay the same

*only change the sign


If the wave function stays the same, there are no problems: the two identical particles can happily stay together and are called bosons. If the wave function changes the sign then we have a problem: we cannot tell which system has the particles swapped because they are identical, so we actually have two differently signed wave functions (which sum to null) for the same system. The solution is not allow such a system: identical particles whose exchange leads to change of sign of the wave function are not allowed to stay together, they are called fermions.
Nature chose electrons as fermions and so you cannot find two identical electrons in the same atom: each electron must have at least one property that allows to distinguish it from all the others, this is called Pauli exclusion principle. Each energy level (determined by the closure of the wave function as John Rennie explained) can hold a limited number of electrons which depends on the complexity of the level. The simplest level is just a sphere and does not offer any way to distinguish two electrons, so it can just hold... two! This is a tiny complication which comes from the spin: an intrinsic property that for electrons can be up or down allowing two of them with opposite spin to stay together on the same level.
A: I can sniff a lot of familiarity with the ''HOW'' answers from what I interpret about you from your post, so I'll only focus on the objective point - ''WHY''.
It turns out that it is possible to meaningfully describe nature by postulating that any object would tend to be in the minimum energy state possible under a given set of physical conditions. So, first we need an understanding of what these minimum energy configurations are - for which we treat the HOWS- (Schrodinger eqn etc.). But once we know what they are, the question is - how do they arrange themselves within these structures, which gets a common sense answer from the Aufbau principle, which is once again the reiteration of the same idea. 
But what is special about this idea becomes clear if you start considering alternatives. Suppose, this wasn't the case, and we chose the stark opposite alternative - every object tended to occupy the most energetic state available (like some bouncing ball collisions in some video games), we would have a really tough time describing nature. For example, we wouldn't be able to explain why any system reaches an equilibrium at all, since e.g. it would be more favorable for an object once set in motion, to keep moving towards an unbounded maximum energy. Now, infinity isn't a number by definition, it reflects an unbounded maximum, so an ''inverted-scale'' description, with infinity in place of 0, is horribly inappropriate. e.g. zero is unique on the number line, but the functions $x$ and $x^2$ both increase indefinitely as we increase $x$. So, ''unbounded from above'' won't be a unique choice, and our description of nature won't be coherent. Anyways, it does so turn out from observational experience that things around us behave as if the underlying principle concerned a minimum, rather than a maximum. So, our postulate seems validated by nature.
Now, to specifically address the question of electronic arrangement, in e.g. Bohr atom, which would make a simple example, introducing the quantization conditions (as you are probably aware of, from your question.) Imagine it this way - if electron is getting attracted to the positively charged nucleus, it will tend to bump into it. However, it doesn't, because of its orbital angular momentum (let us neglect spin for the moment), which will cause it to rotate around the nucleus at some orbital radius, owing to a balance between the centripetal force and this attraction. However, while the electrostatic attraction is a continuous function of $r$, falling off as $1/r^2$, with the quantization condition, the angular momentum can't vary continuously, it grows in discrete units. Thus, the balance condition would now imply that all $r$ aren't allowed. You reach a balance at some fixed values of $r$, which define for you the shell locations (Of course the same condition also gives you the permissible energies.) 
Now, here's the point (and here's how it relates to my -2 voted first paragraph): once you know the permitted energies, you need some filling up principle, and it most convenient to fill them up using our guiding principle of lowest energy first, rather than the other way round. If we filled them the other way round, we can never explain why there should be a hydrogen atom at all, since the very first electron would be sitting infinitely far away from the nucleus and would behave like an ionized, free electron.  
