Thanks if you take the time to read this. Here is the problem statement:
The problem I'm getting is that I'm not getting the kinetic energy diagonal when I convert to the coordinates that diagonalize the potential energy. If you scroll all the way down you should see there's a cross derivative term involving the partial derivatives w.r.t $q_1$ and $q_2$. I haven't been able to spot any significant mistakes. So is there some other way to simultaneously diagonalize potential and the conjugate momenta in the new coordinates? Is there something I'm missing? How does the $\frac{1}{2(2M + m_2)}$ factor come into play?
First we make a transformation in the x coordinates so that the $a$'s go away. Namely $$\begin{bmatrix} x_1^\prime \\x_2^\prime \\ x_3^\prime\end{bmatrix} = \begin{bmatrix} x_1 \\x_2 \\ x_3 \end{bmatrix} - \begin{bmatrix} a/2 \\-a/2 \\ -3a/2\end{bmatrix} $$ And I will refer to the $x_i^\prime$ coordinates without the prime hereafter. $$(x_2 - x_1)^2 + (x_3 - x_2)^2 = x_1^2 +x_2^2 - 2x_1x_2 + x_2^2+ x_3^2 - 2x_2x_3 $$ There are multiple choices for the matrix which below which would produce the above algebraic expression. We want to choose the one which Hermitian, so that the corresponding transformation is unitary. $$= \begin{bmatrix} x_1 &x_2 & x_3\end{bmatrix}\begin{bmatrix} 1 &-1 &0 \\ -1 &2 &-1 \\ 0 & -1 & 1 \end{bmatrix}\begin{bmatrix} x_1 \\x_2 \\ x_3\end{bmatrix} $$ $$det\begin{bmatrix} 1 - \lambda &-1 &0 \\ -1 &2- \lambda &-1 \\ 0 & -1 & 1- \lambda \end{bmatrix} = \left(1 - \lambda \right)^2\left(2 - \lambda \right) - \left(1 - \lambda \right) - \left(1 - \lambda \right) $$ \begin{align*} = \left(1 - \lambda \right)\left[\left(1 - \lambda \right)\left(2 - \lambda \right) -2 \right] &= \left(1 - \lambda \right)\left[2 - 3\lambda + \lambda^2 -2 \right] \\ & =\lambda\left(1 - \lambda \right)\left(\lambda - 3 \right) \end{align*} So the set of eigenvalues is $\left\{0, 1, 3 \right\}$. $$U = \begin{bmatrix} 1/\sqrt{3} &1/\sqrt{3} &1/\sqrt{3} \\ 1/\sqrt{2} &0 &-1/\sqrt{2} \\ 1/\sqrt{6} & -2/\sqrt{6} & 1/\sqrt{6} \end{bmatrix}$$ $$ U^{T}\begin{bmatrix} 0 &0 &0 \\ 0 &1 &0 \\ 0 & 0 & 3 \end{bmatrix}U = \begin{bmatrix} 1 &-1 &0 \\ -1 &2 &-1 \\ 0 & -1 & 1 \end{bmatrix}$$ $$\begin{bmatrix} q_1 \\q_2 \\ q_3\end{bmatrix} = U \begin{bmatrix} x_1 \\x_2 \\ x_3\end{bmatrix}$$ Next begins the calculation of the partial derivatives of $x_i$ in the new coordinate frame, as the first step to finding the KE in terms of the momenta conjugate to the new coordinates $q_i$. $$\begin{bmatrix}\frac{\partial }{\partial x_1} & \frac{\partial }{\partial x_2} & \frac{\partial }{\partial x_3}\end{bmatrix} = \begin{bmatrix}\frac{\partial }{\partial q_1} & \frac{\partial }{\partial q_2} & \frac{\partial }{\partial q_3}\end{bmatrix}U$$ $$KE = -\hbar^2\begin{bmatrix}\frac{\partial }{\partial x_1} & \frac{\partial }{\partial x_2} & \frac{\partial }{\partial x_3}\end{bmatrix} \begin{bmatrix} \frac{1}{2M} &0 &0 \\ 0 &\frac{1}{2m_2} &0 \\ 0 & 0 & \frac{1}{2M} \end{bmatrix} \begin{bmatrix}\frac{\partial }{\partial x_1} \\ \frac{\partial }{\partial x_2} \\ \frac{\partial }{\partial x_3}\end{bmatrix} $$ $$ = -\hbar^2\begin{bmatrix}\frac{\partial }{\partial q_1} & \frac{\partial }{\partial q_2} & \frac{\partial }{\partial q_3}\end{bmatrix}U \begin{bmatrix} \frac{1}{2M} &0 &0 \\ 0 &\frac{1}{2m_2} &0 \\ 0 & 0 & \frac{1}{2M} \end{bmatrix}U^{T}\begin{bmatrix}\frac{\partial }{\partial q_1} \\ \frac{\partial }{\partial q_2} \\ \frac{\partial }{\partial q_3}\end{bmatrix} $$ $$ U \begin{bmatrix} \frac{1}{2M} &0 &0 \\ 0 &\frac{1}{2m_2} &0 \\ 0 & 0 & \frac{1}{2M} \end{bmatrix}U^{T}$$ $$= \begin{bmatrix} 1/\sqrt{3} &1/\sqrt{3} &1/\sqrt{3} \\ 1/\sqrt{2} &0 &-1/\sqrt{2} \\ 1/\sqrt{6} & -2/\sqrt{6} & 1/\sqrt{6} \end{bmatrix}\begin{bmatrix} \frac{1}{2M\sqrt{3}} & \frac{1}{2M\sqrt{2}} &\frac{1}{2M\sqrt{6}} \\ \frac{1}{2m_2\sqrt{3}} & 0 &-\frac{2}{2m_2\sqrt{6}} \\ \frac{1}{2M\sqrt{3}} & -\frac{1}{2M\sqrt{2}} & \frac{1}{2M\sqrt{6}} \end{bmatrix}$$ $$ = \begin{bmatrix} \frac{1}{3M} + \frac{1}{6m_2} & 0 & \frac{1}{3\sqrt{2}}\left(\frac{1}{M} - \frac{1}{m_2} \right) \\ 0 & \frac{1}{2M} & 0 \\ \frac{1}{3\sqrt{2}}\left(\frac{1}{M} - \frac{1}{m_2} \right) & 0 & \frac{1}{6M} + \frac{1}{3m_2} \end{bmatrix} $$ $$\frac{KE}{-\hbar^2} = \begin{bmatrix}\frac{\partial }{\partial q_1} & \frac{\partial }{\partial q_2} & \frac{\partial }{\partial q_3}\end{bmatrix} \begin{bmatrix}\left(\frac{1}{3M} + \frac{1}{6m_2} \right)\frac{\partial }{\partial q_1} + \frac{1}{3\sqrt{2}}\left(\frac{1}{M} - \frac{1}{m_2} \right)\frac{\partial }{\partial q_3}\\ \frac{1}{2M}\frac{\partial }{\partial q_2} \\ \frac{1}{3\sqrt{2}}\left(\frac{1}{M} - \frac{1}{m_2} \right)\frac{\partial }{\partial q_1}+ \left(\frac{1}{6M} + \frac{1}{3m_2} \right)\frac{\partial }{\partial q_3}\end{bmatrix} $$ $$= \left(\frac{1}{3M} + \frac{1}{6m_2} \right)\frac{\partial^2 }{\partial q_1^2} + \frac{1}{2M}\frac{\partial^2 }{\partial q_2^2} + \left(\frac{1}{6M} + \frac{1}{3m_2} \right)\frac{\partial^2 }{\partial q_3^2}$$ $$+ \frac{\sqrt{2}}{3}\left(\frac{1}{M} - \frac{1}{m_2} \right)\frac{\partial^2 }{\partial q_1 \partial q_3} $$ So to me the fact that there is a cross term means that I won't able to express the Hamiltonian in terms of three independent oscillators.