I) OP asks (v5):
Why are the Lagrangian points $L_1$, $L_2$ & $L_3$ unstable?
Actually, that's a great & non-trivial question. One should not make conclusions based on merely the effective potential $V$ (consisting of gravity & the centrifugal potential) alone, because the Coriolis force may be important.
For instance, the global maximum points $L_4$ & $L_5$ of $V$ (which naively would be the last places to suspect a stable equilibrium point), are sometimes stable, cf. e.g. this Phys.SE post!
II) Let us here analyze the stability of an arbitrary point with small initial velocity $v_0$ in the orbital plane$^1$ $\mathbb{R}^2\cong \mathbb{C}$ in the presence of the Coriolis force. Let us pick coordinates $z\in\mathbb{C}$ so that the initial position is in the origin $z=0$.
Let us start by studying critical/stationary points, i.e. Lagrange points. We use the following theorem mentioned in Ref. 1:
Theorem. Given a Hessian ${\bf H}$ for an effective potential $V:\mathbb{R}^2\to \mathbb{R}$ at a Lagrange point. Let $\Omega$ be the angular velocity. The following 3 conditions are necessary & sufficient conditions for the Lagrange point to be stable:
$C~:=~\det({\bf H})~\geq~0. $
$B~:=~{\rm tr}({\bf H})+4\Omega^2~\geq~0. $
$D~:=~B^2-4C~\geq~0. $
NB: The theorem ignores higher-order terms in $V$, which could become important if $\det({\bf H})=0$.
A proof of the theorem is given in my Phys.SE answer here, which also discusses the case of a local maximum.
Case of a local minimum$^2$ $\det({\bf H})\geq 0 \wedge {\rm tr}({\bf H})\geq 0$: It is stable because all 3 conditions are satisfied. To see the third condition, observe that
$$\begin{align} D~=~&B^2-4C~=~({\rm tr}({\bf H})+4\Omega^2)^2-4 \det({\bf H}) \cr
~=~&{\rm tr}({\bf H})-4 \det({\bf H}) + 8{\rm tr}({\bf H})\Omega^2 +16\Omega^4 \cr
~=~&(H_{11}-H_{22})^2 +4H_{12}^2 + 8\underbrace{{\rm tr}({\bf H})}_{\geq 0}\Omega^2 +16\Omega^4 ~>~ 0.\end{align}\tag{1} $$
$\Box$
Case of a strong saddle point $\det({\bf H})<0$: This is the case of the Lagrange points $L_1$, $L_2$ & $L_3$, which OP asks about. It is unstable, because it violates the first condition. $\Box$
III) To gain some intuition for above conclusions, let us also analyze what happens away from the Lagrange points.
Case of a non-critical point: The specific effective force $f=\left. -\nabla V\right|_{z=0}\neq 0$ is non-zero. Then in the region where we can treat the specific effective force as constant, the EOM with specific Coriolis force $2\vec{v}\times\vec{\Omega}$ reads$^3$
$$ \begin{align}\ddot{z} ~\approx~&\underbrace{-2i\Omega \dot{z}}_{\text{Coriolis}} +f\cr
\Updownarrow ~& \cr\frac{d}{dt}\left( \exp\left\{2i\Omega t \right\}\dot{z}\right)~\approx~&\exp\left\{2i\Omega t \right\}f. \end{align}\tag{2}$$
Hence the velocity is
$$ \dot{z}~\approx~\frac{f}{2i\Omega}+ \left(v_0-\frac{f}{2i\Omega}\right) \exp\left\{-2i\Omega t \right\}, \tag{3} $$
and the position is
$$ z~\approx~
\underbrace{\frac{ft}{2i\Omega}}_{\text{drift}}+ \underbrace{\frac{1}{2i\Omega}\left(\frac{f}{2i\Omega}-v_0\right)}_{\text{complex amplitude}} \left(\underbrace{\exp\left\{-2i\Omega t \right\}}_{\text{circle}}-1\right) \tag{4} $$
$$~\approx~ v_0t + \left(f-2i\Omega v_0\right)\frac{t^2}{2}\left(1- \frac{2i\Omega t}{3}\right) + {\cal O}(t^4) . \tag{5} $$
The test particle performs a circular motion with a drift along equipotential lines in the orbital plane. (To the $\color{red}{\rm right}$ if you are looking $\color{red}{\rm downhill}$.) In any case, the test particle is unstable at a non-critical point. $\Box$
$\uparrow $ Fig. 1: A possible horseshoe orbit along equipotential lines.
The above description provides some inside into why local minimums and (sometimes) local maximums are stable, while saddle points are always unstable. If the test particle tends to drift along equipotential lines, then near local minimums and maximums, it ends up in small concentric loops, while equipotential lines always point away from saddle points.
$\uparrow $ Fig. 2: A test particle tends to drift along equipotential lines due to the Coriolis force.
References:
- J. Binney & S. Tremaine, Galactic Dynamics, 2nd edition (2008); p. 181-182.
--
$^1$ There could also be an oscillatory motion perpendicular to the orbital plane, which we ignore.
$^2$ The above analysis is valid for any effective potential $V$. In the restricted 3-body problem, which OP asks about, the effective potential $V$ has no local minimum.
$^3$ Here we choose the orientation of the angular velocity $\vec{\Omega}$ to be prograde/counter-clockwise/positive direction of rotation.