Here is a slightly hand-waving demonstration both that there is an equilibrium point and that motion around it is simple-harmonic for small displacements. Note that it's possible to work this out by actually writing down expressions for the effective torque and approximating them in suitable ways, but those expressions are mildly hairy and I wanted to avoid that since I'm lazy!
Despite this being the accepted answer I now think it is probably wrong in various places (specifically the handwaving around the first-order term from gravity) I will try to improve it.
I'm not very happy with this answer: it started off with me thinking 'I can show this intuitively in a few lines' but then it got away from me and became what it is now. It would perhaps actually be easier to write down the expressions for the effective torque and actually doing the power-series expansions to show it explicitly after all. Sorry about that.
In what follows I'm assuming a very idealised version of the system: friction and other factors are just ignored. So, when I talk about the unstable equilibrium point if you ignore the wire torque I think that would be rather difficult to detect in practice!
The system with only the moving masses
First of all consider the system with only the moving masses -- so the fixed masses are removed. There's now no gravitational attraction. The torque from the suspension wire is $-\kappa(\theta - \theta_0)$ where $\theta_0$ is the equilibrium position (this might be true only for $\theta - \theta_0$ 'reasonably small', but we don't have to worry about that).
Because of the form of the torque it's clear that:
- there is a stable equilibrium ($\theta = \theta_0$);
- the motion of the system around this equilibrium position is SHM.
The system with the static masses added
Now add the two static masses, of mass $M$, and place them so that the centres of the static and moving masses coincide when $\theta = 0$ (this is just a choice of origin of the $\theta$ coordinate really). And make all the masses point masses or sufficiently small spherical masses, just to make things simpler So, what I have to show is:
- for suitable values of $M$ and $\theta_0$ there's still a stable equilibrium position;
- for sufficiently small displacements around this equilibrium position the motion is still SHM.
To do this I'll talk about the torque, $T$, on the rotating system. This is a function of $\theta$ and it has two components: $T(\theta) = T_W(\theta) + T_G(\theta)$: $T_W$ is from the wire (and is given above) and $T_G$ is the gravitational component.
If there is an equilibrium position, say $\theta_E$ then $T(\theta_E) = 0$: the condition for equilibrium is that there must be no net torque.
For the equilibrium position to be stable then in some interval around $\theta_E$, $dT/d\theta \lt 0$. What this means is that if $\theta$ is slightly more than $\theta_E$ then the torque will pull it back & similarly the other way around.
For the motion around $\theta_E$ to be SHM then, in a small interval around $\theta_E$ the torque must look like $T(\theta) = 0 - \alpha(\theta - \theta_E) + \text{higher-order terms}$. So long as $\theta - \theta_E$ is small we can ignore the higher-order terms and then the motion is SHM.
First of all, there is at least one equilibrium position. This can be seen by considering the two limits $\theta \to 0$ and $\theta \to \pi$: these correspond to the cases where the masses are brought very close together at the limits of the swing. In both these cases $T_G$ becomes large and will dominate $T$:
- when $\theta \to 0$ then $T$ is large and negative;
- when $\theta \to \pi$ then $T$ is large and positive.
But $T$ is continuous in $(0, \pi)$ and so there must be at least one zero in that interval. Thus there is at least one equilibrium position: call it $\theta_E$.
So now we can write an expression for the torque at $\theta_E$ in terms of a Taylor expansion of $T_G$ and an explicit expression for $T_W$.
$$
\begin{aligned}
T(\theta) &= T_W(\theta) + T_G(\theta)\\
&= -\kappa(\theta_E - \theta_0) - \kappa(\theta - \theta_E)\\
&\quad + T_G(\theta_E) + T_{G,1}(\theta - \theta_E)
+ T_{G,2}(\theta - \theta_E)^2 + O((\theta - \theta_E)^3))\\
&= -\kappa(\theta_E - \theta_0) + T_G(\theta_E)\\
&\quad + (-\kappa + T_{G,1})(\theta - \theta_E)\\
&\quad + T_{G,2}(\theta - \theta_E)^2\\
&\quad + O((\theta - \theta_E)^3))
\end{aligned}
$$
where $T_{G,n}$ means the $n$th coefficient of the Taylor expansion of $T_G$ about $\theta_E$.
So, OK, since we know this is an equilibrium position we know that $-\kappa(\theta_E - \theta_0) + T_G(\theta_E) = 0$: if that's not true then it's not equilibrium. So we're left with
$$
\begin{aligned}
T(\theta) &= (-\kappa + T_{G,1})(\theta - \theta_E)\\
&\quad + T_{G,2}(\theta - \theta_E)^2\\
&\quad + O((\theta - \theta_E)^3))
\end{aligned}
$$
So this looks almost right: if $(\theta - \theta_E)$ is small then the term in $(\theta - \theta_E)^2$ can be ignored and the motion looks like SHM. The problem is that we don't know what $T_{G,1}$ is: it might be just the right value to cancel out the term from the wire's torque, in which case we can't ignore the $(\theta - \theta_E)^2$ term (and the equilibrium is not stable, unless that term too is zero and the first non-zero term is an odd-order one).
OK, so now the handwavy bit: $T_{G,1} = 0$. Here's why: if you consider the system without the torque from the wire (so make the wire infinitely thin, or use some frictionless bearing or something) then it's clear that there is only an unstable equilibrium position, at $\theta = \pi/2$ (ie with the moving masses at 90 degrees to the static ones). Any perturbation from this position will cause the masses to swing towards each other, one way or the other. That means that there can't be a first-order term for the gravititional torque, because if there was there'd be a stable equilibrium.
So, given that bit of handwaving we can write an expression for $T$ around $\theta_E$ as:
$$
\begin{aligned}
T(\theta) &= -\kappa (\theta - \theta_E)\\
&\quad + O((\theta - \theta_E)^2)
\end{aligned}
$$
And we're done: this is an equilibrium position, it's stable, and motion around it is simple-harmonic for small perturbations. And we can use the period of that motion to determine $\kappa$.