Diagonalization of this Hamiltonian: How do I transform the differential operators? In order to find the eigenstates of this Hamilatonian
$$
H = \sum_{j=1}^3 \left( - \frac{\hbar^2}{2m} \frac{\partial^2}{\partial x_j^2}\right)
+ \frac{1}{2} m \omega^2 \left( (x_1 - x_2)^2 + (x_1 - x_3)^2 + (x_2 - x_3)^2\right)
$$
I want to do a coordinate transformation $y_1 = x_1 - x_2$, $y_2 = x_1 - x_3$, $y_3 = x_2 - x_3$, but I am not sure how to transform the differential operators. Are they just $\frac{\partial^2}{\partial y_j^2}$? Or do they transform to something else? How do I calculate this?
 A: Since you're changing from $x_j$ to $y_j$, you have to consistently make this change to the $x_j$'s everywhere they occur, including in differential operators. To do this, you need to use the Jacobian matrix. The rule can be written like this:
$$\begin{pmatrix}
\frac{\partial}{\partial x_1} \\
\frac{\partial}{\partial x_2} \\
\frac{\partial}{\partial x_3}
\end{pmatrix}
=
\underbrace{
\begin{pmatrix}
\frac{\partial y_1}{\partial x_1} &
\frac{\partial y_2}{\partial x_1} &
\frac{\partial y_3}{\partial x_1} \\
\frac{\partial y_1}{\partial x_2} &
\frac{\partial y_2}{\partial x_2} &
\frac{\partial y_3}{\partial x_2} \\
\frac{\partial y_1}{\partial x_3} &
\frac{\partial y_2}{\partial x_3} &
\frac{\partial y_3}{\partial x_3}
\end{pmatrix}
}_\text{Jacobian matrix}
\begin{pmatrix}
\frac{\partial}{\partial y_1} \\
\frac{\partial}{\partial y_2} \\
\frac{\partial}{\partial y_3}
\end{pmatrix}$$
but if you go through the matrix multiplication, you'll see that this is just three instances of the chain rule. So it's equivalent to solving for the $y_j$'s in terms of the $x_j$'s and then using the chain rule on those formulas.
Note that in some cases, you'll see a change of variables where one or more of the new variables has the same label as one of the original variables. For example, changing Cartesian coordinates $x,y,z$ to cylindrical coordinates $\rho,\theta,z$. It's easy to think you can get away with leaving the common variable(s), $z$ in this case, out of the Jacobian, but you can't. In this situation, I find it best to invent a completely new set of names for the new variables - for example, think of it as changing from $x,y,z$ to $\rho,\theta,q$ - and then when you're done, just rename $q$ to $z$ in the formulas you find.
A: There is no mistake on your exercise sheet.
To find a solution, you have to postulate a linear dependence of the $y^i$ from the $x^j$, $y^i= a^i_j x^j$,where the matrix $a$ of coefficients $a^i_j$ is constant. You have $x^j = (a^{-1})^j_i y^i$.
Now, simply express the initial terms as a function of $y^i$ and express diagonal constraints, you have :
$\sum\limits_{j=1}^3a_j^i a_j^k= \delta ^{ik} \tag{1}$
for the differential term constraint, and :
$\sum\limits_{j=1}^3(a^{-1})^j_i (a^{-1})^{[j+1]}_k= 0 \tag{2}$
for the quadratic $y^2$ term constraint, where $[j+1]=(j+1) \mod 3$
By noting that : $a_j^k = (a^t) _k^j$, and $(a^{-1})^{[j+1]}_k = (a^{(-1) t})_{[j+1]}^k$, $(1)$ and $2$ can be rewritten :
$\sum\limits_{j=1}^3a_j^i (a^t) _k^j= \delta ^i_k\tag{3}$
and :
$\sum\limits_{j=1}^3(a^{-1})^j_i (a^{(-1)t})_{[j+1]}^k= 0 \tag{4}$
The equation $3$ means simply that $aa^t= \mathbb Id$, so $a$ is an orthogonal matrix. For simplicity, we may search for a matrix so that $a=a^t=a^{-1}$, so we have only to check the equation $(4)$, rewritten : 
$\sum\limits_{j=1}^3a^j_i a_{[j+1]}^k= 0 \tag{5}$
A solution is : 
$a = \dfrac{1}{3}\begin {pmatrix} 1&-2&-2\\-2&1&-2\\-2&-2&1 \end {pmatrix}$
