How and why can random matrices answer physical problems? Random matrix theory pops up regularly in the context of dynamical systems.
I was, however, so far not able to grasp the basic idea of this formalism. Could someone please provide an instructive example or a basic introduction to the topic?
I would also appreciate hints on relevant literature.
 A: The basic idea is that statistical properties of complex physical systems
fall into a small number of universal classes. A very known example of
this phenomenon is the universal law implied by the central limit theorem
where the sum of a large number of random variables belonging to a large
class of distrubutions converges to the normal distribution.  Please see
Percy Deift's article for a historical and
motivational review of the subject. Of course, one of the most motivating
examples (mentioned Percy Deift's article) in is the original Wigner's
explanation of the heavy nuclei spectra. Wigner "conjectured" the
universality and looked for a model which can explain the repulsion
between the energy levels of the large nuclei (two close energy levels
are unlikely) which led him to the Gaussian orthogonal ensemble having
this property built in. Now, the heavy nuclei Hamiltonian matrix elements
are not random, but since by universality , for large N, the distribution of the
eigenvalues does not depend on the matrix elements distribution, then the
random matrix eigenvalue distribution approximates that of the
Hamiltonian.
A: David Bar Moshe's answer is fine, but I wanted to go into more detail. The main reason that random matrices show up in dynamical systems is because they describe the level statistics of classically chaotic motions. In classically integrable systems, there is a semiclassical formula for the level-spacing, determined by the Bohr-Sommerfeld rule. If you know the classical energy as a function of the action variables
$$ E(J_1, J_2, J_3...., J_n)$$
you know the quantum energy spacings by setting the J variables to be integer multiples of Planck's constant h.
$$ E(h n_1, h n_2, .... h n_k)$$  
This means that the levels near some reference state are spaced according to the rule:
$$ \Delta E(\Delta n_1,\Delta n_2, ... , \Delta n_k) = {\partial E\over \partial J_i} \Delta n_i $$
And
$$ {\partial E \over \partial J_i} = {2\pi \over T_i} $$
Where $T_i$ is the classical period. This rule means that the levels are distributed more or less uniformly, multiperiodically, as a superposition of different equally spaced points with equally spaced distances between the equally spaced sequences. It's not hard to imagine--- just imagine one period is very long, so that the spacings are very close, and the other period is short, so that the spacing is large, and they you get the energy levels are two uniform interpenetrating sequences of energies which lie on top of each other.
This integrable picture is completely false for complex nuclei, where the energy levels exhibit level repulsion. This means that the levels are not superpositions of equispaced sequences as they are in the classical limit of an integrable system, rather, there must be interactions between the levels that lead them to repel, so they don't want to be close.
In classical mechanics, this phenomenon is the destruction of invariant tori when there are resonances, and this leads large systems to become chaotic. The generic chaotic behavior has universal features, and this is what Wigner discovered. He reasoned that if you are looking at a chaotic Hamitlonian system, the statistics of the levels near a given level are going to be all mixed up in a way that is different from the integrable case. In the integrable case (like the rotations of a rigid molecule, which gives rotation levels superposed on top of excitation spectrum) the spacings tell you something about the periods of the classical motions. But in the chaotic case, there are no periods, and the level details don't tell you anything about the system (at least locally). So Wigner reasoned that if you diagonalize any old matrix with random entries, you will get eigenvalues distributed locally like the physical nuclei.
This is a remarkable true prediction. If you take any old rotation-type matrix, chosen at random from a probability distribution (say the uniform distribution on the group, it's compact), depending only on whether you are talking about a large unitary, real orthogonal, or symplectic matrix, the eigenvalues will be distributed in different places according to a density which depends on the probability distribution you choose, but the local level spacing will have statistics which are indistiguishable from that of chaotic physical systems.
The predictions of this theory have been confirmed by the observation of level repulsion in nuclei. This both confirms that the nuclear motion is classically chaotic (if it has a classical analog) and that random matrix theory describes such chaotic system level statistics.
