Diagonalisation of two coupled Quantum Harmonic Oscillators with different frequencies I was able to diagonalise $H=\hbar\omega a^{\dagger}a+\hbar\omega b^{\dagger} b+\hbar(u a^{\dagger}b+u b^{\dagger} a)$ using  the Bogoliubov transformation \begin{equation}
    a=\cos{\alpha}c_1+\sin{\alpha}c_2,\qquad b=-\sin{\alpha}c_1+\cos{\alpha}c_2
\end{equation}
However, I can't seem to work it out for the case of different frequencies, i.e. for
$$
H=\hbar\Omega a^{\dagger}a+\hbar\omega b^{\dagger} b+\hbar(u^{*} a^{\dagger}b+u b^{\dagger} a)
$$
I have tried to follow a similar procedure by using the Bogoliubov transformation
\begin{equation}
    c_i=u_ia+v_i b, \qquad c_i^{\dagger}=u_i^{*}a^{\dagger}+v_i^{*}b^{\dagger}
\end{equation}
for $i=1,2$ with $[c_i,c_j^{\dagger}]=\delta_{ij},[c_i,c_j]=0$.
Inverting:
$$
\begin{aligned}
a &= (v_2 c_1 - v_1 c_2)\frac{1}{u_1 v_2 - u_2 v_1}=(v_2 c_1 - v_1 c_2)\frac{1}{detUV} \\
b &= (-u_2 c_1 + u_1 c_2)\frac{1}{u_1 v_2 - u_2 v_1}=(-u_2 c_1 + u_1 c_2)\frac{1}{detUV}
\end{aligned}
$$
Then I eventually arrive at
Now substitute these expressions into the Hamiltonian:
\begin{equation}
\begin{aligned}
    H/\hbar=\Omega a^{\dagger}a+\omega b^{\dagger} b+u^{*}a^{\dagger}b+u b^{\dagger} a\\
    =\frac{1}{|detUV|^2}\\
    \Omega(|v_2|^2 c_1^{\dagger}c_1-v_1v_2^{*}c_1^{\dagger}c_2-v_1^{*}v_2c_2^{\dagger}c_1+|v_1|^2c_2^{\dagger}c_2)\\
    +\omega(|u_2|^2c_1^{\dagger}c_1-u_1u_2^{*}c_1^{\dagger}c_2-u_1^{*}u_2c_2^{\dagger}c_1+|u_1|^2c_2^{\dagger}c_2)\\
    +u^{*}(-u_2v_2^{*}c_1^{\dagger}c_1+u_1v_2^{*}c_1^{\dagger}c_2+u_2v_1^{*}c_2^{\dagger}c_1-u_1v_1^{*}c_2^{\dagger}c_2)+u(-u_2^{*}v_2c_1^{\dagger}c_1+u_2^{*}v_1c_1^{\dagger}c_2+u_1^{*}v_2c_2^{\dagger}c_1-u_1^{*}v_1c_2^{\dagger}c_2)\\
    =\frac{1}{|detUV|^2}\\
    (\Omega|v_2|^2+\omega|u_2|^2-u^{*}u_2v_2^{*}-uu_2^{*}v_2)c_1^{\dagger}c_1\\
    +(\Omega|v_1|^2+\omega|u_1|^2-u^{*}u_1v_1^{*}-uu_1^{*}v_1)c_2^{\dagger}c_2\\
    +(-\Omega v_1v_2^{*}-\omega u_1u_2^{*}+u^{*}u_1v_2^{*}+uu_2^{*}v_1)c_1^{\dagger}c_2\\
    +(-\Omega v_1^{*}v_2-\omega u_1^{*}u_2+u^{*}u_2v_1^{*}+uu_1^{*}v_2)c_2^{\dagger}c_1
\end{aligned}
\end{equation}
Assuming this is correct, I don't know how to kill off the off-diagonal terms.
 A: A good start could be to re-write the Hamiltonian as
$$
H=\hbar\Omega a^{\dagger}a+\hbar\omega b^{\dagger} b+\hbar(u^{*} a^{\dagger}b+u b^{\dagger} a)=
\begin{pmatrix}a^\dagger&b^\dagger\end{pmatrix}
\begin{pmatrix}\hbar\Omega&\hbar u^*\\\hbar u&\hbar\omega\end{pmatrix}
\begin{pmatrix}a\\b\end{pmatrix}
$$
Now you can diagonalize the 2-by-2 matrix and express the new operators in terms of its eigenvectors. E.g., we could introduce transformation matrix $S$, satisfying $S^\dagger=S^{-1}$ a, $\det S=1$, so that the new operators are
$$
\begin{pmatrix}a\\ b\end{pmatrix}=S\begin{pmatrix}c_1\\ c_2\end{pmatrix}\Leftrightarrow
\begin{pmatrix}c_1\\ c_2\end{pmatrix}=S^\dagger\begin{pmatrix}a\\ b\end{pmatrix}
$$
and
$$
S^\dagger
\begin{pmatrix}\hbar\Omega&\hbar u^*\\\hbar u&\hbar\omega\end{pmatrix}
S=
\begin{pmatrix}\hbar\omega_1&0\\ 0&\hbar\omega_2\end{pmatrix}
$$
Note that simple form
$$
S=\begin{pmatrix}\cos\alpha&\sin\alpha\\ -\sin\alpha&\cos\alpha\end{pmatrix}
$$
works only for real matrix. In case of complex values of $u,u^*$ one needs to use a more general expression.
Remark: Term Bogoliubov transformation is usually applied to transformation that mixes the annihilation and creation operators, as we do in case of superconductors. In this case I would simply call it just a canonical transformation or transformation of basis/rotation in Hilbert space.
