David Bar Moshe's answer is fine, but I wanted to go into more detail. The main reason that random matrices show up in dynamical systems is because they describe the level statistics of classically chaotic motions. In classically integrable systems, there is a semiclassical formula for the level-spacing, determined by the Bohr-Sommerfeld rule. If you know the classical energy as a function of the action variables
$$ E(J_1, J_2, J_3...., J_n)$$
you know the quantum energy spacings by setting the J variables to be integer multiples of Planck's constant h.
$$ E(h n_1, h n_2, .... h n_k)$$
This means that the levels near some reference state are spaced according to the rule:
$$ \Delta E(\Delta n_1,\Delta n_2, ... , \Delta n_k) = {\partial E\over \partial J_i} \Delta n_i $$
And
$$ {\partial E \over \partial J_i} = {2\pi \over T_i} $$
Where $T_i$ is the classical period. This rule means that the levels are distributed more or less uniformly, multiperiodically, as a superposition of different equally spaced points with equally spaced distances between the equally spaced sequences. It's not hard to imagine--- just imagine one period is very long, so that the spacings are very close, and the other period is short, so that the spacing is large, and they you get the energy levels are two uniform interpenetrating sequences of energies which lie on top of each other.
This integrable picture is completely false for complex nuclei, where the energy levels exhibit level repulsion. This means that the levels are not superpositions of equispaced sequences as they are in the classical limit of an integrable system, rather, there must be interactions between the levels that lead them to repel, so they don't want to be close.
In classical mechanics, this phenomenon is the destruction of invariant tori when there are resonances, and this leads large systems to become chaotic. The generic chaotic behavior has universal features, and this is what Wigner discovered. He reasoned that if you are looking at a chaotic Hamitlonian system, the statistics of the levels near a given level are going to be all mixed up in a way that is different from the integrable case. In the integrable case (like the rotations of a rigid molecule, which gives rotation levels superposed on top of excitation spectrum) the spacings tell you something about the periods of the classical motions. But in the chaotic case, there are no periods, and the level details don't tell you anything about the system (at least locally). So Wigner reasoned that if you diagonalize any old matrix with random entries, you will get eigenvalues distributed locally like the physical nuclei.
This is a remarkable true prediction. If you take any old rotation-type matrix, chosen at random from a probability distribution (say the uniform distribution on the group, it's compact), depending only on whether you are talking about a large unitary, real orthogonal, or symplectic matrix, the eigenvalues will be distributed in different places according to a density which depends on the probability distribution you choose, but the local level spacing will have statistics which are indistiguishable from that of chaotic physical systems.
The predictions of this theory have been confirmed by the observation of level repulsion in nuclei. This both confirms that the nuclear motion is classically chaotic (if it has a classical analog) and that random matrix theory describes such chaotic system level statistics.