Why do we think there are only three generations of fundamental particles? In the standard model of particle physics, there are three generations of quarks (up/down, strange/charm, and top/bottom), along with three generations of leptons (electron, muon, and tau).  All of these particles have been observed experimentally, and we don't seem to have seen anything new along these lines.  A priori, this doesn't eliminate the possibility of a fourth generation, but the physicists I've spoken to do not think additional generations are likely.

Question: What sort of theoretical or experimental reasons do we have for this limitation?

One reason I heard from my officemate is that we haven't seen new neutrinos.  Neutrinos seem to be light enough that if another generation's neutrino is too heavy to be detected, then the corresponding quarks would be massive enough that new physics might interfere with their existence.  This suggests the question: is there a general rule relating neutrino masses to quark masses, or would an exceptionally heavy neutrino just look bizarre but otherwise be okay with our current state of knowledge?
Another reason I've heard involves the Yukawa coupling between quarks and the Higgs field.  Apparently, if quark masses get much beyond the top quark mass, the coupling gets strong enough that QCD fails to accurately describe the resulting theory.  My wild guess is that this really means perturbative expansions in Feynman diagrams don't even pretend to converge, but that it may not necessarily eliminate alternative techniques like lattice QCD (about which I know nothing).
Additional reasons would be greatly appreciated, and any words or references (the more mathy the better) that would help to illuminate the previous paragraphs would be nice.
 A: One part of the answer to this question is that the neutrinos are Majorana particles (or Weyl--- the two are the same in 4d), which can only acquire mass from nonrenormalizable corrections. The neutrinos do not have a right handed partner in an accessible energy range. If there is such a partner, it is very very heavy. So this means that they have to be exactly massless if the standard model is exactly renormalizable.
The interactions that give neutrinos mass are two-Higgs two-Lepton scattering events in the standard model Lagrangian, where the term is $HHLL$ with the SU(2) indices of each H contracted with an L. This term gives neutrino masses, but is dimension 5, so is suppressed by the natural energy scale, which is 1016 GeV, the GUT scale. This gives the measured neutrino masses. This term also rules out a low energy Planck scale.
If you have another generation, the next neutrino would have to be light, just because of this suppression. There is no way to couple the Higgs to the next neutrino much stronger than the other three. There are only 3 light neutrinos, as revealed by the Z width, BBN, as others said.
A: There are very good experimental limits on light neutrinos that have the same electroweak couplings as the neutrinos in the first 3 generations from the measured width of the $Z$ boson. Here light means $m_\nu < m_Z/2$. Note this does not involve direct detection of neutrinos, it is an indirect measurement based on the calculation of the $Z$ width given the number of light neutrinos. Here's the PDG citation: 
http://pdg.lbl.gov/2010/listings/rpp2010-list-number-neutrino-types.pdf
There is also a cosmological bound on the number of neutrino generations coming from production of Helium during big-bang nucleosynthesis. This is discussed in "The Early Universe" by Kolb and Turner although I am sure there are now more up to date reviews. This bound is around 3 or 4.
There is no direct relationship between quark and neutrino masses, although you can derive
possible relations by embedding the Standard Model in various GUTS such as those based on
$SO(10)$ or $E_6$. The most straightforward explanation in such models of why neutrinos are light is called the see-saw mechanism 
http://en.wikipedia.org/wiki/Seesaw_mechanism
and leads to neutrinos masses $m_\nu \sim m_q^2/M$ where $M$ is some large mass scale on the order of $10^{11} ~GeV$ associated with the vacuum expectation value of some Higgs field that plays a role in breaking the GUT symmetry down to $SU(3) \times SU(2) \times U(1)$. If the same mechanism is at play for additional generations one would expect the neutrinos to be lighter than $M_Z$ even if the quarks are quite heavy. 
Also, as you mentioned, if you try to make fourth or higher generations very heavy you have to increase the Yukawa coupling to the point that you are outside the range of perturbation  theory. These are rough theoretical explanations and the full story is much more complicated  but the combination of the excellent experimental limits, cosmological bounds and theoretical expectations makes most people skeptical of further generations. Sorry this wasn't mathier.
A: If instead of "why do we...", you had asked "why do I...", speculative answers could be considered too. In 25 years, I have thought some ones; perhaps some people would like to add more, here as community wiki (doesn't generate rep)  or in the comments if rep is less than 100.


*

*Lock between colour and flavour.

*mass matrix would need to be 3x3 for some reason.

*mass matrix is 3x3 as a minimum to violate CP (but it could be more, then)

*mass matrix is 3x3 in order to use the involved lengths in some discretization of the calculus of derivatives up to second order. Related to the ambiguity of choosing an ordering when quantising terms such as $xp$.

*three generations come from the the relationship between bosonic strings, with 24 transversal directions, and superstrings, with 8. Related to Leech lattice, heterotic strings, etc.

*three generations without the neutrinos are 84 helicities, or three generations with Right and Left neutrinos but excluding the top quark are also 84 helicities. This is also the number of components of the source of the 11D Membrane, of M-theory fame.

*three generations is the only solution for my petty theory, the sBootstrap, to work with leptons besides quarks. And even only with quarks, any other solution is uglier.

A: Theoretical reasons for three generations.
Traditional.
A. Anything less than 3 generations could not introduce CP violation into heavy quark decay.
This actually lead to the prediction of the bottom and top quarks.
GUT/String theory
B. The biggest special lie group is E8, this happens to nicely split into three copies of E6,
producing 3 generations.
A: My research involves a geometric model of spin-1/2 particles, though the discussion of the three generations is beyond the scope of my thesis.  However, if I can figure out how to mention this speculation in the Future Work section at the end of my thesis, I will probably do so.
I can't help but marvel at the coincidence of the number three for generations as well as for dimensions of space (where the inertial reference frame fixes the time dimension related to the spatial dimensions).  If spin was treated as an oscillation (not just an "intrinsic angular momentum"), then higher-generational particles could have more complicated modes of oscillation:  second- and third-generation particles could have two- and three- dimensional spin modes, respectively.  If spin was somehow related to mass (which the magnetic dipole moment seems to say it is), then the greater masses of the higher-generational particles could be explained by these higher-dimensional oscillations.  Somehow.  :)
I am only putting this idea out because I don't suspect I will have the chance to investigate it myself in a more thorough manner.  But who knows, maybe I will, and maybe your comments on the idea will help me hone it.  Or maybe someone else will take it and run with it, which is fine with me as long as I am mentioned in the credits somewhere. ;)
A: Gell-Mann's Baryon Decuplet can get enhanced and it can be shown that the 3 Upper Quarks and the 3 lower Quarks are exactly the points of Gravity of the 6 Triangels. There ist no more Place.
There are also 6 Gluons double colored on the horizontal 1 Spin boson circle, but the multicolored 2 Gluons take the perpendicular places W+1 & W-1.
YouTube.com. Konstruktion Standardmodell
A: As the Gluons don't have any charge, the position of Gluon 7 and 8 takes the free Spinaxis on the PionLeft and Right   answered Oct 20   Hsch31
