Coherent State, Unitary Operators, Harmonic Oscillator Consider the operator:
$$O = e^{\theta(a^\dagger b - b^\dagger a)}$$
where $\theta$ is a constant.
$O$ is a unitary operator.
$a$, $a^\dagger$, $b$, and $b^\dagger$ are ladder operators for two harmonic oscillators.
A normalized coherent state is defined as:
$$\lvert\alpha\rangle = e^{-\lvert\alpha\rvert^2/2} e^{\alpha a^\dagger} \lvert 0\rangle$$
where $\lvert0\rangle$ is the ground state of the harmonic oscillator.
I'm trying to see how $O$ acts on the coherent states by calculating $O \lvert\psi\rangle = O\lvert\alpha\rangle\lvert\beta\rangle$ in terms of coherent states.
Also, how does $O$ act on $\alpha$ and $\beta$?
I'm trying to use 
$$O a O^\dagger = a \cos(\theta) + b \sin(\theta)$$ 
and 
$$O b O^\dagger = -a \sin(\theta) + b \cos(\theta).$$
 A: There are many ways to go around this. You can start from the coherent states and apply the unitary $\hat{O}$ directly on them. That will not be that simple because you will get a term $\hat{O}e^{\alpha\hat{a}^\dagger}e^{\beta\hat{b}^\dagger}$. Now, the typical approach would be to exchange the order of the operators to get something like $e^{\alpha\hat{a}^\dagger}e^{\beta\hat{b}^\dagger}\hat{O}$ (up to some extra term due to the commutator). This not really a simple task but once you are done, you can Taylor expand the operator $\hat{O}$ and keep only the zeroth order (all other terms contain annihilation operators acting on vacuum). I am not going to dig into the calculation in more detail, there are many ways to do it and none of them is really pleasant.
But there is a better way. You can define a displacement operator by the action $\hat{D}(\alpha)|0\rangle = |\alpha\rangle$ and then you have $\hat{D}(\alpha)\hat{a}\hat{D}^\dagger(\alpha) = \hat{a}+\alpha$. You can combine this with the formulas for $\hat{O}\hat{a}\hat{O}^\dagger$, $\hat{O}\hat{b}\hat{O}^\dagger$ to see how the annihilation operators are transformed. What you should get is a beam-splitting of the two coherent states, i.e.,
$$|\alpha,\beta\rangle\to|t\alpha+r\beta,t\beta-r\alpha\rangle$$,
where $t = \cos\theta$, $r = \sin\theta$.
A: Let us change OP's notation $a\to a_1$ and $b \to a_2$. We write collectively the two annihilation operators as a column two-vector 
$$ \tag{1} \vec{a}~:=~\begin{bmatrix} a_1 \\ a_2 \end{bmatrix}.$$ 
We have the Heisenberg algebra 
$$ \tag{2} [a_i,a_j^{\dagger}] ~=~\delta_{ij} {\bf 1}, \qquad [a_i,a_j] ~=~0, \qquad [a_i^{\dagger},a_j^{\dagger}] ~=~0,\qquad i,j~\in~\{1,2\}, $$
and the vacuum state 
$$ \tag{3} a_i | 0\rangle ~=~0.  $$
Define un-normalized coherent states 
$$ \tag{4} |\vec{\alpha} )_a ~:=~ e^{ a^{\dagger}_i \alpha_i} | 0\rangle . $$
The idea is now to diagonalize the ${\cal O}$ operator.
Define unitary matrix
$$ \tag{5} U ~:=~\frac{\sqrt{2}}{2} \begin{bmatrix} 1 & i \\ i & 1 \end{bmatrix}~=~ \exp\left[i\frac{\pi}{4}\sigma_x \right] .$$ 
Define new operators
$$ \tag{6} b_i ~:=~ U_{ij} a_j, \qquad [b_i,b_j^{\dagger}] ~=~\delta_{ij} {\bf 1}, $$
and new coherent continuous labels
$$ \tag{7} \beta_i  ~:=~ U_{ij} \alpha_j. $$
Define un-normalized coherent states 
$$ \tag{8} |\vec{\beta} )_b ~:=~ e^{ b^{\dagger}_i \beta_i} | 0\rangle~=~  e^{ a^{\dagger}_i \alpha_i} | 0\rangle~=~|\vec{\alpha} )_a . $$
Note that the operator becomes diagonal 
$$ \tag{9} {\cal O}~:=~ \exp\left[i\theta a^{\dagger}_i (\sigma_y)_{ij} a_j\right]  ~=~ \exp\left[i\theta b^{\dagger}_i (\sigma_z)_{ij} b_j\right]     ~=~ \exp\left[i\theta (n_1-n_2)\right],$$
where the number operators read
$$ \tag{10} n_i~:=~b^{\dagger}_i b_i \qquad\text{(no sum over $i$).}  $$
Next deduce the commutation relations
$$ \tag{11}  \exp\left[i\theta n_i\right]\exp\left[b^{\dagger}_i \beta_i \right]~=~\exp\left[b^{\dagger}_i \beta_i e^{i\theta} \right]\exp\left[i\theta n_i\right] \qquad\text{(no sum over $i$).} $$
We conclude from (11) that
$$ \tag{12} {\cal O}|\beta_1, \beta_2 )_b  ~=~|\beta_1e^{i\theta}, \beta_2e^{-i\theta} )_b. $$
We leave it as an exercise to translate (12) back to normalized $a$-coherent states. 
A: Actually the operator you have is closely related to angular momentum.    Indeed you can verify that the operators
$$
b^\dagger a \mapsto L_+\, ,\qquad
a^\dagger b\mapsto L_-\, ,\qquad
\frac{1}{2}(b^\dagger b-a^\dagger a) \mapsto L_z
$$
satisfy the same commutation relations as the angular momentum operators.   In this notation, the lowest state of angular momentum $s$ is the 2d harmonic oscillator state
$$
\frac{(a^\dagger)^{2s}}{\sqrt{(2s)!}}\vert 0\rangle\to \vert s,-s\rangle
$$
and in particular 
$$
a\vert 0\rangle \mapsto \vert
\textstyle \frac{1}{2},-\frac{1}{2}\rangle\, ,\\
b\vert 0\rangle\mapsto 
\textstyle \frac{1}{2},\frac{1}{2}\rangle\, .
$$
In general
$$
\frac{(a^\dagger)^{s-m} (b^\dagger)^{s+m}}{\sqrt{(s-m)!(s+m)!}}\vert 0\rangle \mapsto \vert s,m\rangle\, .
$$
Thus your operator
$$
a^\dagger b-b^\dagger a\mapsto L_--L_+
=L_x-iL_y-(L_x+iLy)=-2iL_y
$$
so that you can think of $O$ as the rotation $e^{-2i\theta L_y}$.  
Now, the coherent states
\begin{align}
\vert\alpha\rangle \vert\beta\rangle&=
\sum_{p,q}\frac{\alpha^p \beta^q}{p! q!}
(a^\dagger)^p(b^\dagger)^q\vert 0\rangle \\
&\mapsto 
\sum_{p,q}\frac{\alpha^p \beta^q}{p! q!}
\vert \textstyle\frac{1}{2}(p+q),\frac{1}{2}(p-q)\rangle \\
\end{align}
where $\vert \textstyle\frac{1}{2}(p+q),\frac{1}{2}(p-q)\rangle$ is an angular momentum state with $s=
\textstyle\frac{1}{2}(p+q)$ and $m_s=\frac{1}{2}(p-q)\rangle$.  Thus
\begin{align}
O\vert\alpha\rangle \vert\beta\rangle
&\mapsto 
\sum_{sm_s}\frac{\alpha^{s-m_s} \beta^{s+m_s}}{(s+m_s)! (s-m_s)!}
e^{-i2\theta L_y}\vert \textstyle s,m_s\rangle\, ,\\
&=\sum_{sm_s}\frac{\alpha^{s-m_s} \beta^{s+m_s}}{(s+m_s)! (s-m_s)!}
\sum_{m_s'} \vert s,m_s'\rangle d^s_{m_s',m_s}(2\theta)
\end{align}
where $d^s_{m_s',m_s}(2\theta)$ is 
a Wigner $d$-function.
It remains to convert back $\vert s,m_s'\rangle$ to harmonic oscillator states.
Finally, the solution simplifies nicely if your initial state is either
$\vert 0\rangle \vert\beta\rangle$ or $\vert \alpha \rangle \vert 0\rangle$.  In such cases the functions $d^s_{m_s',\pm s}(2\theta)$ have a reasonably simple form, containing no summation.
