Three topics:
- There's an electromagnetic analogy for the non-inertial forces in a rotating reference frame.
- Before looking at the restricted three-body problem of the question, analyze the familiar two-body problem in a rotating reference frame. I think that interpreting these results in terms of the known solutions for this problem is instructive.
- Finally, look at the Lagrange points.
1. Electromagnetic analogy.
It turns out that the non-inertial forces on a mass $m$ in a rotating reference frame can be manipulated into exactly the form of the electromagnetic Lorentz force on a charge $q$ in an inertial frame.
Reference by Moreno and Barrachina (pdf) . There are some unfortunate typos, but I think the results I state are accurate.
Specifically, for a rotating reference frame with angular velocity $\boldsymbol{\omega}$, if the electromagnetic scalar potential $\phi$ and vector potential $\boldsymbol{A}$ are taken to be:
$$ \phi = -\left(\frac{m}{q}\right) \frac{\boldsymbol{(\omega \times r)^2}}{2}$$
$$ \boldsymbol{A} = \left(\frac{m}{q}\right) \boldsymbol{\omega \times r} $$
then the electromagnetic force on the charge $q$ in an inertial frame is the same as the non-inertial forces on the mass $m$ in the rotating frame.
In particular, for a constant angular velocity $\boldsymbol{\omega}$ that is perpendicular to the plane of motion (like in this problem), in cylindrical coordinates $(r,\theta,z)$, the analogous electric field $\boldsymbol{E}$ and magnetic field $\boldsymbol{B}$ are:
$$E=\left(\frac{m}{q}\right) \omega^2 r \, \, \boldsymbol{\hat{r}}$$
$$B=\left(\frac{m}{q}\right) 2 \omega \boldsymbol{\hat{z}}$$
The electric field "simulates" the centrifugal force, and the magnetic field the Coriolis force. All intuition from the Lorentz force then carries over: the Coriolis force diverts a body perpendicular to its velocity, forming circles in the absence of other forces.
2. Two-body problem in a rotating frame.
The two-body problem ($M_s$ (sun), $M_e$ (earth)) can be formulated in a rotating frame with angular velocity $\omega$ (making the angular velocity the independent variable). Performing the usual conversion to a single-body potential $U$, one gets:
$$U=-\frac{GM_t M_r}{r} - M_r \frac{\omega^2 r^2}{2} -M_r \omega r^2 \dot{\theta}$$
$$\text{ where } \qquad M_t=M_s+M_e , \qquad \frac{1}{M_r}=\frac{1}{M_s} + \frac{1}{M_e} , \qquad
\alpha=\frac{M_e}{M_t}, \beta=\frac{M_s}{M_t} $$
Solving for a stationary point $r_o$ (with $\dot{\theta}=0$), one gets the usual result:
$$r_o = \left(\frac{G M_t}{\omega^2} \right)^\frac{1}{3} ,
\text{sun x-y coordinates }(x_s,y_s)=(-\alpha r_o, 0) , \text{earth }(x_e,y_e)=(\beta r_o, 0)$$
But observe that the potential at $r_o$ is a maximum, not a minimum!
To see what's going on, find the linearized equations of motion around the stationary point.
$$(x,y)=(x_o+\delta x,y_o+\delta y) \qquad , \qquad (v_x,v_y) = (\delta v_x, \delta v_y) $$
$$ \frac{d}{dt}
\left( \begin{array}{c}
\delta x \\ \delta y \\ \delta v_x \\ \delta v_y
\end{array} \right)
= \left( \begin{array}{cccc}
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 1 \\
-q_{xx} & -q_{xy} & 0 & 2 \omega \\
-q_{xy} & -q_{yy} & -2 \omega & 0
\end{array} \right)
\left( \begin{array}{c}
\delta x \\ \delta y \\ \delta v_x \\ \delta v_y
\end{array} \right) $$
$$ \text{ with } q_{xx}=\frac{1}{m} \frac{d^2U}{dx^2}, q_{yy}=\frac{1}{m} \frac{d^2U}{dy^2}, \text{ and } q_{xy}=\frac{1}{m} \frac{d^2U}{dx \, dy} $$
The natural frequencies $s_i$ are the roots of the characteristic polynomial, which turns out to be:
$$ s^4 - \Gamma s^2 + \Delta = 0$$
$$ \text{ where } \Gamma = -( q_{xx} + q_{yy} +4 \omega^2)
\text{ and } \Delta = q_{xx} q_{yy} - q_{xy}^2$$
For this case, only $q_{xx}=-3 \omega^2$ is non-zero (negative curvature at a maximum), so one finds $\Gamma =- \omega^2$ and $\Delta=0$, and the natural frequencies are:
$$ s_{1,2} = 0 \qquad , \qquad s_{3,4}=\pm i \omega $$
corresponding to neutral response or oscillatory motion at the rotation frequency. The Coriolis force bends the trajectories and thereby "converts" an unstable equilibrium into a center of oscillatory motion.
Interpretation: A circular orbit (corresponding to the stationary point) is only one possible solution of the two-body problem; in general orbits are ellipses, so one should expect oscillatory motion around the stationary point, periodic at the rotation frequency. (I think the 0 natural frequencies correspond to an azimuthal shift in position.) The point is that the motion is not unstable, despite it being a local maximum, and the lack of damping (negative real part of the natural frequencies) should be expected.
(By the way, if $q_{xx}$ were positive, corresponding to a local minimum, the resulting oscillatory natural frequencies would be at a higher frequency that $\omega$, and the resulting motion would not be elliptical.)
3. Lagrange points
The restricted three-body problem can be analyzed in the same fashion as above, and the stability of each stationary point determined. A reference is Cornish (pdf). It turns out that L1, L2, and L3 all have positive real natural frequencies and therefore are unstable. L4 and L5 have all pure imaginary natural frequencies (as long as $M_e<25 M_s$), and therefore have motions similar to that of the two-body problem.
Update: Here are some notes on the character of the dynamics at the various Lagrange points:
L4 & L5: These points are most like the two-body problem, with a relatively small, negative (de-stabilizing) potential curvature that is stabilized by the Coriolis force.
L3: This point (planet "x" on the other side of the sun from the earth) is the most interesting to me. Since the curvatures are again small (like for the earth two body problem), and one of them is in fact positive (stabilizing), it seems odd that L3 is unstable. The resolution is that the opposite-sign curvatures form a saddle point. A body can escape from such a point by traveling slowly near the line of zero curvature (and hence zero force); with the proper angle and velocity it accelerates slowly and the Coriolis force is ineffective at stopping the escape.
L1 & L2: These points are unstable on two counts: a) the large negative (de-stabilizing) potential curvature from the nearby earth overwhelms the stabilizing effect of the Coriolis force and b) the principal curvatures differ in sign (like L3), forming a saddle point.