$\let\om=\omega \def\qt{{\textstyle {1 \over 4}}}
\def\half{{\textstyle {1 \over 2}}}$
Here is the analytical treatment. Let $x_1, x_2, x_3$ be the
displacements of the three masses from their equilibrium positions.
$k$ are spring constants. Then the forces acting are:
- on mass 1: $F_1 = k\,(x_2 - 2\,x_1)$
- on mass 2: $F_2 = k\,(x_1 + x_3 - 2\,x_2)$
- on mass 3: $F_3 = k\,(x_2 - 2\,x_3)$.
There are 3 normal modes, easily identified by symmetry:
- mode $a$: all masses oscillating in phase, $m_1$ and $m_2$ with equal
amplitude, $m_3$ with a possibily different amplitude (we shall see
its amplitude is greater)
- mode $b$: $m_1$ and $m_3$ oscillating in opposition, with equal
amplitudes; $m_2$ stationary
- mode $c$: like $a$ but $m_2$ oscillates in opposition.
In equations:
Mode $a$
$$x_1 = x_3 = a_1 \cos\om_a t \qquad x_2 = a_2 \cos\om_a t \quad
(a_1, a_2 > 0).\tag1$$
Mode $b$
$$x_1 = -x_3 = b\,\cos\om_b t \qquad x_2 = 0.\tag2$$
Mode $c$
$$x_1 = x_3 = c_1 \cos\om_c t \qquad x_2 = c_2 \cos\om_c t \quad
(c_1 > 0,\; c_2 < 0).\tag3$$
So you see that equations for modes $a$ and $c$ are the same apart
signs for $a_2$, $c_2$. Actually we'll find both in one shot.
Note that eqs. (1), (2), (3) assume all initial velocities are zero.
Otherwise additional terms with $\sin\om_a t$ etc. would have been
needed.
Applying $F=ma$ we get for each mode a system of three equations.
Mode $a$
$$m\,\ddot x_1 = k\,(x_2 - 2\,x_1)$$
$$m\,\ddot x_2 = k\,(x_1 + x_3 - 2\,x_2)$$
(the third equation is useless).
$$-m\,\om_a^2 a_1 = k\,(a_2 - 2\,a_1) \tag4$$
$$-m\,\om_a^2 a_2 = k\,(2\,a_1 - 2\,a_2).$$
Dividing
$${a_1 \over a_2} = {a_2 - 2\,a_1 \over 2\,a_1 - 2\,a_2}$$
$$a_1 (2\,a_1 - 2\,a_2) = a_2 (a_2 - 2\,a_1)$$
$$2\,a_1^2 = a_2^2$$
$$a_2 = \sqrt2\,a_1.$$
Mode $c$ gives the same equations but we must take
$$c_2 = -\sqrt2\,c_1.$$
We may use (4) to find $\om_a$ and $\om_c$:
$$-m\,\om_a^2 a_1 = k\,a_1\,(\sqrt2 - 2)$$
$$\om_a = \sqrt{(2 - \sqrt2)\,{k \over m}}$$
$$\om_c = \sqrt{(2 + \sqrt2)\,{k \over m}}.$$
Mode $b$
$$m\,\ddot x_1 = -2\,k\,x_1$$
$$-m\,\om_b^2 b = -2\,k\,b$$
$$\om_b = \sqrt{2k \over m}.$$
Let's summarize
$$\om_a = \sqrt{(2 - \sqrt2)\,{k \over m}} \qquad
\om_b = \sqrt{2k \over m} \qquad
\om_c = \sqrt{(2 + \sqrt2)\,{k \over m}}.$$
Mode $a$:
$$x_1 = a\,\cos\om_a t \qquad x_2 = a \sqrt2\,\cos\om_a t \qquad
x_3 = a\,\cos\om_a t$$
Mode $b$:
$$x_1 = b\,\cos\om_b t \qquad x_2 = 0 \qquad x_3 = -b\,\cos\om_b t$$
Mode $c$:
$$x_1 = c\,\cos\om_c t \qquad x_2 = -c \sqrt2\,\cos\om_c t \qquad
x_3 = c\,\cos\om_c t.$$
General solution (with $\dot x_1(0) = \dot x_2(0) = \dot x_3(0) = 0$)
$$\eqalign{
x_1(t) &= a\,\cos\om_a t + b\,\cos\om_b t + c\,\cos\om_c t \cr
x_2(t) &= a\,\sqrt2\,\cos\om_a t - c\,\sqrt2\,\cos\om_c t \cr
x_3(t) &= a\,\cos\om_a t - b\,\cos\om_b t + c\,\cos\om_c t.\cr}$$
Note: I followed a step-by-step approach, but a more direct and
general way exists, valid for any number of balls. This post is
already too long, however...
A particular solution
To get the solution satisfying $x_1(0) = x_2(0) = x_3(0) = 1$ we have
to find $a$, $b$, $c$ such that
$$\eqalign{
a + b + c &= 1 \cr
(a - c)\,\sqrt2 &= 1 \cr
a - b + c &= 1 \cr}$$
i.e.
$$a = {2 + \sqrt2 \over 4} \qquad b = 0 \qquad
c = {2 - \sqrt2 \over 4}.$$
Then
$$\eqalign{
x_1(t) = x_3(t) &= {2 + \sqrt2 \over 4}\,\cos\om_a t +
{2 - \sqrt2 \over 4}\,\cos\om_c t \cr
x_2(t) &= {1 + \sqrt2 \over 2}\,\cos\om_a t +
{1 - \sqrt2 \over 2}\,\cos\om_c t.\cr}$$
Here are graphs:
http://www.sagredo.eu/temp/ball-spring-1.eps
Another solution
If $x_1(0) = 1 \ x_2(0) = x_3(0) = 0$ then
$$\eqalign{
a + b + c &= 1 \cr
(a - c)\,\sqrt2 &= 0 \cr
a - b + c &= 0 \cr}$$
i.e.
$$a = c = \qt \qquad b = \half.$$
Then
$$\eqalign{
x_1(t) &= \qt \cos\om_a t + \half \cos\om_b t +
\qt \cos\om_c t \cr
x_2(t) &= {1 \over \sqrt2}\,(\cos\om_a t - \cos\om_c t).\cr
x_3(t) &= \qt \cos\om_a t - \half \cos\om_b t +
\qt \cos\om_c t.\cr}$$
Here are graphs:
http://www.sagredo.eu/temp/ball-spring-2.eps