Finding the maximum extension of a Spring 
I have solved that after the body m1 hits m2, the velocity of m2 is going to be (3/4)*v0. I did this by using the law of conservation of momentum and using the coefficient of restitution (relative velocities). I do not know how to proceed to get the max extension in the spring.
 A: You have found the velocity of body 2 after it is hit by the body 1 absolutely correctly. So let's consider the second part of the process, lasting from the moment of the hit until the moment of the maximum extension of a spring.
Let's put that at initial time, velocity of the body 2 was: $$v_{0,a} = \frac{3}{4}v_0$$
Appropriately, velocity of the body 3 was zero: $v_{0,b} = 0$.
We need to determine somehow when the maximum spring extension would happen. The extension of the spring would be maximum, at the moment when the distance between the bodies is maximum. If the distance is maximum, then the separation(closing) speed is zero, it means that the angular velocity of the two bodies would be equal.
Let's denote the initial and the final angular velocities of the bodies as follows:
$$\omega_{0,a} = \frac{v_{0,a}}{2R} = \frac{3}{4\cdot 2R}v_0\tag{1}$$
$$\omega_{0,b} = 0$$
$$\omega_{1,a} = \omega_{1,b} = \omega_{1}$$
For the system of two bodies 2 and 4, the total angular momentum is the sum of angular momentum of each body:
$$L=I_a\omega_a + I_b\omega_b$$
where $I_a$ and $I_b$ is the moment of inertia of the body 2 and 3 appropriately. 
For the initial time:
$$L_0=I_a\omega_{0,a}$$
For the final time:
$$L_1=(I_a + I_b)\omega_1$$
From the law of conservation of momentum, we know that $L_0 = L_1$, thus:
$$\omega_1 = \frac{I_a}{I_a + I_b}\omega_{0,a}\tag{2}$$
Let's use the law of conservation of energy. First, let's find the energy of the system at the initial time:
$$E_0=\frac{  I_a {\omega_{0,a}}^{2}  }   {2}$$
Energy of the system at final time:
$$E_1= U + \frac{  I_a + I_b } {2}{\omega_1}^{2} $$
where $U$ - the potential energy of extended spring.
Thus, from the law of conservation of energy:
$$U = \frac{  I_a {\omega_{0,a}}^{2}  }   {2}   -  \frac{  I_a + I_b } {2}{\omega_1}^{2}$$
Let's express the $\omega_1$ in terms of $\omega_{0,a}$ by using the equation $(2)$:
$$U = \left(\frac{  I_a   } {2} - \frac{1}{2}\frac{  {I_a}^{2} } {I_a + I_b} \right) {\omega_{0,a}}^{2}\tag{3}$$
From the definition of the moment of inertia, we can get, that:
$$I_a = 4mR^{2} $$
$$I_b = mR^{2} \tag{4}$$
Let's substitute the equations $(4)$ into equation $(3)$:
$$U = \frac{2}{5}mR^{2} {\omega_{0,a}}^{2}\tag{5}$$
Let's express the $\omega_{0,a}$ in terms of $v_0$ by using the equation $(2)$:
$$U = \frac{3^{2}}{4^{2}\cdot2\cdot5}m {v_0}^{2}\tag{6}$$
The potential energy of extended spring is:
$$U=\frac{k{x}^{2}}{2}$$
where $x$ - is the extention of the spring and $k$ - is the stiffness of the spring. Thus, the extension would be:
$$x=\sqrt{\frac{2U}{k}}\tag{7}$$
Eventually, substitute equation $(6)$ into the equation $(7)$:
$$x=\frac{3}{4}v_0\sqrt{\frac{m}{5k}}$$
So, the correct answer is B
