First, to clarify: I'm not asking if perhaps there's a moon that we haven't found yet. The question is, theoretically, would the earth be able to have another stable moon in addition to the current one? Or, if the orbit couldn't be stable, why not? How large/small of a moon would it be able to have? And how do we know all this?
Jupiter has over 60 moons, and the dozens of man-made satellites (along with the thousands of pieces of space-trash) orbiting the Earth could be considered tiny moons.
Earth's main Moon would disrupt the orbit of anything smaller at certain radii. The particular disrupted orbits are called resonances, and occur where the orbital period in question divided by the period of the Moon reduces to a small fraction, like 1/2, 2/3, etc. It comes from the Moon giving a predictable tug on objects at those resonance orbits, similar to pushing a child on a swing higher and higher so that you eventually pitch him right out onto the ground. (Do not try this at home.)
The principle is spectacularly illustrated by Saturn's rings. (Some of*) the dozens of dark tracks indicate where one of the moons has a resonance. http://upload.wikimedia.org/wikipedia/commons/thumb/b/b1/Saturn%27s_rings_dark_side_mosaic.jpg/2200px-Saturn%27s_rings_dark_side_mosaic.jpg
[EDIT: To answer more of your questions, there are lots of stable orbits, and the Moon can also act to stabilize, not just destabilize orbits- see http://en.wikipedia.org/wiki/Trojan_(astronomy) . Any stable orbit would have to be above the top of the atmosphere, about 100 km, or else friction with atmospheric gas would drag it down to Earth. Moons could range from specks of dust up to potentially extremely large, even as large as the first Moon, but at great size the options for stability become much more limited. See http://en.wikipedia.org/wiki/Euler%27s_three-body_problem. All of this arises from classical, Newtonian mechanics. To greatly simplify history, Tycho Brahe took a lot of data on planet positions in the sky, Kepler took those observations and developed his purely empirical laws of orbital motion, and Newton took those empirical laws and developed a theoretical framework to account for them.]
*Some gaps are due to other or unknown factors, but for the most part...
|show 1 more comment|