What is the meaning of this hamiltonian? I have a problem (homework) with this hamiltonian:
$H=\epsilon \left( a^{\dagger}a + b^{\dagger}b \right) + g \left( a^{\dagger}b^{\dagger} + ba \right)$
All over the books I am searching, I was unable of finding something like this.
The problem is:


What means to have two different sets of creation-annihilation operators? How they affect each other?
Using the suggested transformation yields to many terms like $\alpha^{\dagger}\beta$, $\beta^{\dagger}\alpha$, and so on.
What should I do here? 
 A: Indeed you have two sets of creation and annihilation operators that are related by the Bogoliubov transformation. 
The maths
If you rewrite $H$ as as a function of $\alpha,\beta,\alpha^\dagger and \beta^\dagger$ you should find something like:
$$H = \epsilon(u^* u+v^* v)( \alpha^\dagger \alpha + b^\dagger \beta) + g((u^*u^* - vv)\alpha^\dagger b^\dagger + (uu-v^*v^*)\beta \alpha + (u^*v^* + vu)(\alpha^\dagger \alpha + \beta^\dagger\beta))$$
Where I have dropped some normal ordering terms (basically this means that I neglect the Casimir effect) for this to be diagonal we need:
$$\langle\psi|H|\phi\rangle \sim \langle\psi|\phi\rangle $$ the terms that spoil this are the $\alpha^\dagger \beta^\dagger$ and $\alpha \beta$ terms such they must be set to zero as was already suggested in the question. Said terms vanish if $u=\pm v$ let us set $u=v$ for definiteness, this is irrelevant since in the end we are only interested in expectation values and those are quadratic...
So ok, now we know that:
$$a = \frac{1}{\sqrt2}(\alpha - \beta^\dagger)$$
$$b = \frac{1}{\sqrt2}(\beta + \alpha^\dagger)$$
And our new Hamiltonian is diagonal.
The interpretation
Supposedly you found your original Hamiltonian in some physical context and the original set of creation and annihilation operators must have been some excitation of that system. We found that it is much more convenient to rewrite the Hamiltonian in function of some other exitations since it then becomes diagonal.
e.g. original set may have been left running and right running waves. It appears that this does not diagonalize the Hamiltonian. However left+right running and left-right running does diagonalize it.
Some more math
There is one very interesting feature about this that you will probably handle soon after this namely the dependence of the vacuum. We may now define our vacuum to mean:
$a|0\rangle = b|0\rangle = 0$ which is the vacuum with respect to the diagonal excitations. Rewriting this vacuum in terms of the original set of particles yields:
$\alpha |0\rangle = \beta^\dagger |0\rangle$ and $\beta |0\rangle = -\alpha^\dagger |0\rangle$ which is solved by:
$$|0\rangle = C_{te}\exp(-\alpha^\dagger \beta^\dagger)$$
This tells us that the vacuum for the new particle excitations is full of the non diagonal excitations ! This is very similar to the Hawking-effect where it is found that the black hole vacuum contains many particles when it is observed by and observer that is in flat spacetime...
I hope you gained some insight, feel free to ask questions in the comments.
*Disclaimer: I might have been a bit quick with commutation relations i.e. and missed some signs. The ideas will be correct though :) 
For example, I didn't know your commutation relations to begin with and I switched operators quite a few times. This may have introduced sign errors if I assumed wrong commutations... I assumed anticommuation*
