How did the Cavendish experiment to measure $G$ work? I'm really confused about the Cavendish experiment.

What I'm confused about is why the rod would oscillate around the equilibrium point. 
(From here on, I'm calling the balls at the end of the spinning rod "balls" and the really big masses that are causing the gravitational attraction "heavy fixed masses")
As far as I understand, the equilibrium point is where the net torque on the contraption is zero, since the torque provided by the gravitational force from the heavy fixed masses and the torque provided by the torsion of the twisted wire balance out.
However, lets say that the balls go slightly beyond the equilibrium point. 
If the gravitational force (and therefor the torque provided by the heavy fixed masses) was a constant, then yes, I agree that the balls would oscillate around the equilibrium point.
But the gravitational force is not constant! It increases as the distance between the balls and the heavy fixed masses decreases, inversely proportionally to the distance between the balls and the heavy fixed masses squared!
What's to say that the gravitational force doesn't increase faster than the torsion force increases, and the balls simply crash into the heavy fixed masses?
 A: The force of gravity between the two balls is a descending curve starting from large values, but not infinity, because the minimum distance at contact of the two balls is $ R_{distance }=(r_1+r_2) \ $,  to gently approaching zero at $R_{distance} = \infty \quad$ but always stays positive.
But the force of torsion of the wire is linear, at small angles, and can revers sign and is guaranteed to become smaller than F, when the wire unwinds back and force.
Suppose the system is initially balanced such that even when the balls touch $ T\ > F \quad, \  $ F being gravity force and T, torque.
So the torque will swing the two balls back, while decreasing in value such that at certain $R_{distance}  , \ $T becomes smaller than F  and gravity pulls the bells back, repeating the cycle.
And the value of $ \ F - T \ , $ will swing between positive and negative, causing the oscillation of the two balls.
A: @Joshua Ronis  As I said the torque of the wire swings back an forth. Changing sign from clockwise to counter clockwise, When it passes the wire's equilibrium center.
Even though the gravity is small but the torque is now in addition to it and works at the same direction as gravity to bring the 2 balls back near the mass, but then changes sign again and fights the gravity. 
Try to imagine the two balls swinging even in the absence of the big mass. 
A: Here is a slightly hand-waving demonstration both that there is an equilibrium point and that motion around it is simple-harmonic for small displacements.  Note that it's possible to work this out by actually writing down expressions for the effective torque and approximating them in suitable ways, but those expressions are mildly hairy and I wanted to avoid that since I'm lazy!
Despite this being the accepted answer I now think it is probably wrong in various places (specifically the handwaving around the first-order term from gravity) I will try to improve it.
I'm not very happy with this answer: it started off with me thinking 'I can show this intuitively in a few lines' but then it got away from me and became what it is now.  It would perhaps actually be easier to write down the expressions for the effective torque and actually doing the power-series expansions to show it explicitly after all.  Sorry about that.
In what follows I'm assuming a very idealised version of the system: friction and other factors are just ignored.  So, when I talk about the unstable equilibrium point if you ignore the wire torque I think that would be rather difficult to detect in practice!
The system with only the moving masses
First of all consider the system with only the moving masses -- so the fixed masses are removed.  There's now no gravitational attraction.  The torque from the suspension wire is $-\kappa(\theta - \theta_0)$ where $\theta_0$ is the equilibrium position (this might be true only for $\theta - \theta_0$ 'reasonably small', but we don't have to worry about that).
Because of the form of the torque it's clear that:


*

*there is a stable equilibrium ($\theta = \theta_0$);

*the motion of the system around this equilibrium position is SHM.


The system with the static masses added
Now add the two static masses, of mass $M$, and place them so that the centres of the static and moving masses coincide when $\theta = 0$ (this is just a choice of origin of the $\theta$ coordinate really).  And make all the masses point masses or sufficiently small spherical masses, just to make things simpler  So, what I have to show is:


*

*for suitable values of $M$ and $\theta_0$ there's still a stable equilibrium position;

*for sufficiently small displacements around this equilibrium position the motion is still SHM.


To do this I'll talk about the torque, $T$, on the rotating system.  This is a function of $\theta$ and it has two components: $T(\theta) = T_W(\theta) + T_G(\theta)$: $T_W$ is from the wire (and is given above) and $T_G$ is the gravitational component.
If there is an equilibrium position, say $\theta_E$ then $T(\theta_E) = 0$: the condition for equilibrium is that there must be no net torque.
For the equilibrium position to be stable then in some interval around $\theta_E$, $dT/d\theta \lt 0$. What this means is that if $\theta$ is slightly more than $\theta_E$ then the torque will pull it back & similarly the other way around.
For the motion around $\theta_E$ to be SHM then, in a small interval around $\theta_E$ the torque must look like $T(\theta) = 0 - \alpha(\theta - \theta_E) + \text{higher-order terms}$.  So long as $\theta - \theta_E$ is small we can ignore the higher-order terms and then the motion is SHM.
First of all, there is at least one equilibrium position.  This can be seen by considering the two limits $\theta \to 0$ and $\theta \to \pi$: these correspond to the cases where the masses are brought very close together at the limits of the swing.  In both these cases $T_G$ becomes large and will dominate $T$:


*

*when $\theta \to 0$ then $T$ is large and negative;

*when $\theta \to \pi$ then $T$ is large and positive.


But $T$ is continuous in $(0, \pi)$ and so there must be at least one zero in that interval.  Thus there is at least one equilibrium position: call it $\theta_E$.
So now we can write an expression for the torque at $\theta_E$ in terms of a Taylor expansion of $T_G$ and an explicit expression for $T_W$.
$$
\begin{aligned}
 T(\theta) &= T_W(\theta) + T_G(\theta)\\
           &= -\kappa(\theta_E - \theta_0) - \kappa(\theta - \theta_E)\\
           &\quad + T_G(\theta_E) + T_{G,1}(\theta - \theta_E)
              + T_{G,2}(\theta - \theta_E)^2 + O((\theta - \theta_E)^3))\\
           &= -\kappa(\theta_E - \theta_0) + T_G(\theta_E)\\
           &\quad + (-\kappa + T_{G,1})(\theta - \theta_E)\\
           &\quad + T_{G,2}(\theta - \theta_E)^2\\
           &\quad + O((\theta - \theta_E)^3))
\end{aligned}
$$
where $T_{G,n}$ means the $n$th coefficient of the Taylor expansion of $T_G$ about $\theta_E$.
So, OK, since we know this is an equilibrium position we know that $-\kappa(\theta_E - \theta_0) + T_G(\theta_E) = 0$: if that's not true then it's not equilibrium.  So we're left with
$$
\begin{aligned}
 T(\theta) &= (-\kappa + T_{G,1})(\theta - \theta_E)\\
           &\quad + T_{G,2}(\theta - \theta_E)^2\\
           &\quad + O((\theta - \theta_E)^3))
\end{aligned}
$$
So this looks almost right: if $(\theta - \theta_E)$ is small then the term in $(\theta - \theta_E)^2$ can be ignored and the motion looks like SHM.  The problem is that we don't know what $T_{G,1}$ is: it might be just the right value to cancel out the term from the wire's torque, in which case we can't ignore the $(\theta - \theta_E)^2$ term (and the equilibrium is not stable, unless that term too is zero and the first non-zero term is an odd-order one).
OK, so now the handwavy bit: $T_{G,1} = 0$.  Here's why: if you consider the system without the torque from the wire (so make the wire infinitely thin, or use some frictionless bearing or something) then it's clear that there is only an unstable equilibrium position, at $\theta = \pi/2$ (ie with the moving masses at 90 degrees to the static ones).  Any perturbation from this position will cause the masses to swing towards each other, one way or the other. That means that there can't be a first-order term for the gravititional torque, because if there was there'd be a stable equilibrium.
So, given that bit of handwaving we can write an expression for $T$ around $\theta_E$ as:
$$
\begin{aligned}
 T(\theta) &= -\kappa (\theta - \theta_E)\\
           &\quad + O((\theta - \theta_E)^2)
\end{aligned}
$$
And we're done: this is an equilibrium position, it's stable, and motion around it is simple-harmonic for small perturbations. And we can use the period of that motion to determine $\kappa$.
