Bogoliubov transformation is not unitary transformation, correct? To diagonalize quadratic term in the antiferromagnet Heisenberg model, we may introduce the Bogoliubov transformation:$a_k=u_k\alpha_k+v_k\beta_k^\dagger$,   $b_k^\dagger=v_k\alpha_k+u_k\beta_k^\dagger$.
This transformation can diagonalize the quadratic term in the Hamiltonian:
\begin{align}
H
&=\sum_k(a^\dagger_ka_k+b^\dagger_kb_k+\gamma_ka^\dagger_kb^\dagger_k+\gamma_ka_kb_k)
\\ & =\sum_{\bf{k}} 
\begin{pmatrix}a_{\bf{k}}^\dagger & b_{\bf{k}}\end{pmatrix}
\begin{pmatrix}1 &\gamma_{\bf{k}}\\\gamma_{\bf{k}} & 1\end{pmatrix}
\begin{pmatrix}a_{\bf{k}} \\ b_{\bf{k}}^\dagger\end{pmatrix}
\\ & =\sum_{\bf{k}} 
\begin{pmatrix}\alpha_{\bf{k}}^\dagger & \beta_{\bf{k}}\end{pmatrix}\begin{pmatrix}u_k &v_{{k}}\\v_{k} & u_{k}\end{pmatrix}
\begin{pmatrix}1 &\gamma_{\bf{k}}\\\gamma_{\bf{k}} & 1_{\bf{k}}\end{pmatrix}\begin{pmatrix}u_k &v_{{k}}\\v_{k} & u_{k}\end{pmatrix}
\begin{pmatrix}\alpha_{\bf{k}} \\ \beta_{\bf{k}}^\dagger\end{pmatrix}
\\ & =\sum_{\bf{k}} 
\begin{pmatrix}\alpha_{\bf{k}}^\dagger & \beta_{\bf{k}}\end{pmatrix}
\begin{pmatrix}\epsilon_k &0\\0 &\epsilon_k\end{pmatrix}
\begin{pmatrix}\alpha_{\bf{k}} \\ \beta_{\bf{k}}^\dagger\end{pmatrix}
\end{align}
with $\epsilon_k=\sqrt{1-\gamma_k^2},u_k=\sqrt{\frac{1+\epsilon_k}{2\epsilon_k}},v_k=-\frac{\gamma_k}{\sqrt{2\epsilon_k(1+\epsilon_k)}}$.
But the transformation U: $\begin{pmatrix}u_k &v_{{k}}\\v_{k} & u_{k}\end{pmatrix}$ is not unitary, because $u_k,v_k$ are real, $U^\dagger\neq U^{-1}$.
Is the number of bosons not conserved, so the transformation may not be unitary? Are there any restriction on the transformation of boson?
 A: You are correct, Bogoliubov transformations are not unitary in general. By definition,

Bogoliubov transformations are linear transformations of creation/annihilation operators that preserve the algebraic relations among them.

The algebraic relations are mainly the commutation/anticommutation relations which define the bosonic/fermionic operators. Nowhere in the definition did we specified that the transformation should be unitary. In fact, the Bogoliubov transformation (in its most generic form) is symplectic for bosons and orthogonal for fermions. In neither case is the Bogoliubov transformation unitary. The Bogoliubov transformation of bosons correspond to the linear canonical transformation of oscillators in classical mechanics (because bosons are quanta of oscillators), and we know the linear canonical transformations are symplectic due to the symplectic structure of the classical phase space.
So to be more specific, what are the restrictions on Bogoliubov transformations? Let us consider the case of $n$ single particle modes of either bosons $b_i$ or fermions $f_i$ (where $i=1,2,\cdots,n$ labels the single particle states, such as momentum eigenstates). Both $b_i$ and $f_i$ are not Hermitian operators, which are not quite convenient for a general treatment (because we can't simply treat $b_i$ and $b_i^\dagger$ as the independent basis since they are still related by the particle-hole transformation). Therefore we choose to rewrite the operators as the following linear combinations (motivated by the idea of decomposing a complex number into two real numbers like $z=x+\mathrm{i}y$):
$$\begin{split}b_i&=a_i+\mathrm{i}a_{n+i}\\b_i^\dagger&=a_i-\mathrm{i}a_{n+i}\end{split}\qquad
\begin{split}f_i&=c_i+\mathrm{i}c_{n+i}\\f_i^\dagger&=c_i-\mathrm{i}c_{n+i}\end{split}$$ 
where $a_i=a_i^\dagger$ and $c_i=c_i^\dagger$ (for $i=1,2,\cdots,2n$) are Hermitian operators (analogus to real numbers). They must inherit the commutation or anticommutation relations from the "complex" bosons $b_i$ and fermions $f_i$:
$$\begin{split}[b_i,b_j^\dagger]=\delta_{ij},[b_i,b_j]=[b_i^\dagger,b_j^\dagger]=0&\Rightarrow[a_i,a_j]=\frac{1}{2}g_{ij}^a \\
\{f_i,f_j^\dagger\}=\delta_{ij}, \{f_i,f_j\}=\{f_i^\dagger,f_j^\dagger\}=0&\Rightarrow\{c_i,c_j\}=\frac{1}{2}g_{ij}^c\end{split}$$
where $g_{ij}^a$ and $g_{ij}^c$ are sometimes called the quantum metric for bosons and fermions respectively. In matrix forms, they are given by
$$g^a=\mathrm{i}\left[\begin{matrix}0&\mathbb{1}_{n\times n}\\-\mathbb{1}_{n\times n}&0\end{matrix}\right] \qquad g^c=\left[\begin{matrix}\mathbb{1}_{n\times n}&0\\0&\mathbb{1}_{n\times n}\end{matrix}\right],$$
with $\mathbb{1}_{n\times n}$ being the $n\times n$ identity matrix. So to preserve the algebraic relations among the creation/annihilation operators is to preserve the quantum metric. General linear transformations of the operators $a_i$ and $c_i$ take the form of
$$a_i\to \sum_{j}W_{ij}^a a_j\qquad c_i\to \sum_{j}W_{ij}^c c_j,$$
where the transformation matrix elements $W_{ij}^a, W_{ij}^c\in\mathbb{R}$ must be real, in order to ensure that the operators $a_i$ and $c_i$ remain Hermitian after the transformation. Then to preserve the quantum metric is to require
$$W^a g^a W^{a\intercal}= g^a\qquad W^c g^c W^{c\intercal}= g^c.$$
So any real linear transformation satisfying the above conditions is a Bogoliubov transformation in the most general sense. Then depending on the property of the quantum metric, the Bogoliubov transformation is either symplectic or orthogonal. For the bosonic quantum metric, $g^a=-g^{a\intercal}$ is antisymmetric, so the transformation $W^a$ is symplectic. For the fermionic quantum metric, $g^c=g^{c\intercal}$ is symmetric, so the transformation $W^c$ is orthogonal. 
A: Unitarity of a quantum mechanical transformation is not determined by how it mixes creation and annihilation operators. (It doesn't matter what kind of matrix—orthogonal, symplectic, or unitary—is involved in the mixing!) Rather, one should examine whether the transformation is associated with a unitary operator acting on the Hilbert space.
The Bogoliubov transformation OP cited can be represented as follows ($\textbf{k}$-dependence is suppressed):
$$
\hat{a} \ \ \rightarrow \ \ \hat{a}^{\prime} =\, \cosh\lambda\, \hat{a} \,+\, \sinh\lambda\,\hat{b}^{\dagger}, \\
\hat{b}^{\dagger}\ \ \rightarrow\ \ \hat{b}^{\prime\,\dagger} =\, \sinh\lambda\, \hat{a} \,+ \,\cosh\lambda\,\hat{b}^{\dagger},
$$
where $\lambda$ is a real number. This transformation is unitary if and only if there exists a unitary operator $U$ such that
$$
\hat{a}^{\prime} = U \hat{a} U^{-1},\\
\hat{b}^{\prime\,\dagger} = U \hat{b}^{\dagger} U^{-1}.
$$
Indeed, these relations are fulfilled with the following choice:
$$
U = \exp\Big[\lambda(\hat{a}\hat{b} - \hat{b}^{\dagger}\hat{a}^{\dagger})\Big],
$$
so the transformation is unitary.
A: Let me work on this part of the matrix equation
$$H=\sum_{\bf{k}} 
\begin{pmatrix}a_{\bf{k}}^\dagger & b_{\bf{k}}\end{pmatrix}
\begin{pmatrix}1 &\gamma_{\bf{k}}\\\gamma_{\bf{k}} & 1\end{pmatrix}
\begin{pmatrix}a_{\bf{k}} \\ b_{\bf{k}}^\dagger\end{pmatrix}=\sum_{\bf{k}} 
\begin{pmatrix}\alpha_{\bf{k}}^\dagger & \beta_{\bf{k}}\end{pmatrix}\begin{pmatrix}u_k &v_{{k}}\\v_{k} & u_{k}\end{pmatrix}
\begin{pmatrix}1 &\gamma_{\bf{k}}\\\gamma_{\bf{k}} & 1_{\bf{k}}\end{pmatrix}\begin{pmatrix}u_k &v_{{k}}\\v_{k} & u_{k}\end{pmatrix}
\begin{pmatrix}\alpha_{\bf{k}} \\ \beta_{\bf{k}}^\dagger\end{pmatrix}
$$
The important part is that the transformation of the fields can be seen as well as a transformation of the matrix 
$$
\Gamma~=~\begin{pmatrix}1 &\gamma_{\bf{k}}\\\gamma_{\bf{k}} & 1\end{pmatrix}~\rightarrow~\begin{pmatrix}u_k &v_{{k}}\\v_{k} & u_{k}\end{pmatrix}
\begin{pmatrix}1 &\gamma_{\bf{k}}\\\gamma_{\bf{k}} & 1_{\bf{k}}\end{pmatrix}\begin{pmatrix}u_k &v_{{k}}\\v_{k} & u_{k}\end{pmatrix}~=~M^\dagger\Gamma M,
$$
where $M^\dagger~=~M$. The determinant of this is $det(M\Gamma M)~=~det(M)det(\Gamma)det(M)$ $=~det(\Gamma)$ The determinant of $M$ then gives $u_k^2~-~v_k^2~=~1$. These can then be represented by $u_k~=~sinh(k)$ and $v_k~=~cosh(k)$.
Now evaluate the commutator $[a_k,~a^\dagger_k]$
$$
[a_k,~a^\dagger_k]~=~u_k^2[\alpha_k,~\alpha_k^\dagger]~+~v_k^2[\beta^\dagger_k,~\beta_k]~=~u_k^2[\alpha_k,~\alpha_k^\dagger]~-~v_k^2[\beta_k,~\beta^\dagger_k].
$$
For the commuators $[\alpha_k,~\alpha_k^\dagger]~=~[\beta_k,~\beta_k^\dagger]~=~1$ and we then see $[a_k,~a_k^\dagger]~=~1$. The same clearly holds $[b_k,~b_k^\dagger]~=~1$ This means that any system with $N\hbar$ units of action is constant. There is no change in the phase space volume of the system. this then means Bogoliubov transformations are effectively unitary.
A: No, it's unitary transformation, but only when you consider the Hamiltonian's electron & hole together. 
