I solved this problem by painful calculations of Lorentz matrices. However, I heard that there is a much easier solution using the generators of boosts and rotations and their commutation relations, plus the Baker-Campbell-Hausdorff identity. How is this possible? Could anyone please show me?
As I indicate in my comment, the statement is made plausible virtually by inspection upon the CBH expansion of the boosts' product and the leading orders of the generators' commutators, but it is cumbersome to carry through getting an explicit answer; in our case determination of the Wigner angle θ and the direction and magnitude of the boost w.r.t. $|v_3| (\hat{x} \cos\phi +\hat{y} \sin\phi)$.
Nevertheless, yes, thanks to Weyl, if you work with the spinor map, the logic of the boost and rotation generators does indeed not only determine these parameters neatly, but further reveals by inspection that the right-hand-side you have is possible and virtually inevitable, before too much explicit algebra.
The secret is that all BCH expansions are easily done explicitly through Pauli matrix algebra and rapidity space parameters in the exponents are ultimately neater. The price to pay is refamiliarizing yourself with the language. (cf, eg, Misner, Thorne, Wheeler, §41.3.)
Pre-Appendix with the language used: Given the isomorphism of the Lorentz group to PSL(2,C), the spinor map takes 4-vectors to 2×2 Hermitian matrices spanned by the Pauli matrices and the identity, $$X = \left[ \begin{matrix} t+z & x-iy \\ x+iy & t-z \end{matrix} \right]=\mathbb{1} t+\vec{\sigma}\cdot \vec{x}. $$ In this space, the Lorentz transformation, a unimodular matrix, acts both on the left and the right, conjugately, $$ X \mapsto \Lambda X \Lambda^\dagger ,$$ preserving Hermiticity.
Moreover, the Lorentz generators are not all Hermitian. Those of the rotations are Hermitian but the ones of the boosts are antihermitian! $$ \vec{K}=\frac{i}{2} \vec{\sigma} \qquad \mapsto B(v_1 \hat{x}) =e^{\zeta \sigma_1/2}, \qquad B(v_2 \hat{y}) =e^{\xi \sigma_2/2},\\ \vec{J}=\frac{1}{2} \vec{\sigma} \qquad \mapsto R(\theta \hat{z})= e^{i\theta \sigma_3/2} . $$ Note the parameters in the exponent are angles and rapidities, $v/c=\tanh \xi$, so $\gamma=\cosh \xi$. These have superior algebraic properties.
Finally, call the direction of the final resultant boost $(\hat{x} \cos\phi +\hat{y} \sin\phi)$ for some angle φ to be determined; so, $\sigma_f=(\hat{x} \cos\phi +\hat{y} \sin\phi)\cdot \vec{\sigma}$; and its parameter/rapidity $f=\operatorname{arctanh} (v_3/c)$.
Further, recall the straightforward expansions of the exponentials of Pauli matrices and Pauli vectors.
As a reminder crib sheet from a practical review book by Başkal, Kim & Noz, use their table on p I-4, Table 1.1.
Now, your boost composition simply reduces to $$ B(v_2 \hat{y})B(v_1 \hat{x}) = e^{\xi \sigma_2/2} e^{\zeta \sigma_1/2}= \left(\mathbb{1} \cosh \frac{\xi}{2} +\sigma_2 \sinh \frac{\xi}{2}\right)\left (\mathbb{1} \cosh \frac{\zeta}{2} +\sigma_1 \sinh \frac{\zeta}{2}\right ) \\ =\mathbb{1} \cosh \frac{\xi}{2} \cosh \frac{\zeta}{2} +i\sigma_3 \sinh \frac{\xi}{2} \sinh \frac{\zeta}{2}+\sigma_2 \sinh \frac{\xi}{2} \cosh \frac{\zeta}{2} +\sigma_1 \sinh \frac{\zeta}{2} \cosh \frac{\xi}{2} . $$
You immediately see that the i which has emerged out of the Pauli matrix multiplication dictates a rotation, the Wigner rotation, in the z direction.
Moreover, $J_3$ corresponding to it will be rotating the x and y boosts into each other. It makes sense, then, to postulate a right-hand side of the form you were given, and to just solve for the unknowns, if possible, $$ B(c \tanh f ~(\hat{x} \cos\phi +\hat{y} \sin\phi) )R_z(\theta) = e^{f \sigma_f/2} e^{i\theta \sigma_3/2}= \left(\mathbb{1} \cosh \frac{f}{2} +\sigma_f \sinh \frac{f}{2}\right) \left(\mathbb{1} \cos \frac{\theta}{2} +i\sigma_3 \sin \frac{\theta}{2}\right)\\ =\mathbb{1} \cosh \frac{f}{2}\cos \frac{\theta}{2} + i\sigma_3 \sin \frac{\theta}{2}\cosh \frac{f}{2} + \sigma_x \cos \left (\phi+\frac{\theta}{2} \right )\sinh \frac{f}{2}+\sigma_y \sin \left (\phi+\frac{\theta}{2} \right )\sinh \frac{f}{2} . $$
And nothing else. So one may solve for θ,φ, and f by comparing this expression with the above, and that's it! Let me solve for θ, to note something rarely appreciated.
Comparing the coefficients of the identity and $\sigma_3$ yields $$\cosh \frac{\xi}{2}\cosh \frac{\zeta}{2}=\cosh \frac{f}{2}\cos \frac{\theta}{2},\\ \sinh \frac{\xi}{2}\sinh \frac{\zeta}{2}=\cosh \frac{f}{2}\sin \frac{\theta}{2}.$$
Divide the second by the first, to get a simple expression for the Wigner angle, $$ \tan \frac{\theta}{2}= \tanh \frac{\xi}{2}\tanh \frac{\zeta}{2}. $$
Through a miracle of trigonometry-cum-hyperbolic-trigonometry, this expression is equivalent to the somewhat mystical angle expression of the other answer,
$$\tan \theta= \frac{\sinh \xi \sinh \zeta }{\cosh \xi +\cosh \zeta}~~,$$
which appears more magical.
This is a standard feature of working in half-angles in rapidity space—the math loves them.
You may also see that $$ \tan \left(\phi+ \frac{\theta}{2}\right )= \tanh \frac{\xi}{2}\coth \frac{\zeta}{2}~, $$ and $$ \cosh^2\frac{f}{2}=\cosh\left( \frac{\xi+\zeta}{2}\right) ~. $$
Finally, you might think that I have been handling group elements and not generators here, but a moment's reflection could point out that it is the rapidities and the angles, so Lie algebra parameters, that flow naturally through the machinery, and not the group objects and parameters. The shortcuts live in the algebra.
