How do you compute the commutator of rotation matrices in two different directions by different angles?
Let $R_{x}(\alpha)$ be the rotation matrix about the $x$-axis and $R_{z}(\beta)$ be the rotation matrix about the $z$ -axis. Now the commutator of $$[R_{x}(\alpha),R_{z}(\beta)]$$ follows the same rule as the commutator of the generators of rotations, like $$[J_{x},J_{z}]=i\hbar J_{y}$$.
Can the commutator of group elements follow the same rules as that of generators? Please clarify.
"The same rule" is associativity, but everything else is quite different.
That is, the group commutator is, in some convention, $$[R_{x}(\alpha),R_{z}(\beta)]= R_{x}(\alpha)R_{z}(\beta) R_{x}(-\alpha)R_{z}(-\beta), $$ where $R_{x}(\alpha)= e^{-i\alpha J_x}$, etc... For notational simplicity, let's call $-i\alpha J_x =A$ and $-i\beta J_z =B$, antihermitean, both. Now, A and B are Lie algebra elements, so their commutator, is a completely different object, $$[A,B]=AB-BA.$$
The two commutators, group and algebra, are, of course, related, $$ e^A e^B e^{-A} e^{-B} \\ =\ \exp\!\left( [A, B] + \frac{1}{2!}[A{+}B, [A, B]] + \frac{1}{3!} \left(\frac{1}{2} [A, [B, [B, A]]] + [A{+}B, [A{+}B, [A, B]]]\right) + \cdots\right), $$ albeit in a messy way. However, as @mike stone points our in his comment, a vanishing algebra commutator $[A,B]=0$ trivializes the group commutator to the identity.
So, to evaluate the group commutator, you only need apply the formula for composing two finite rotations, published by Olinde Rodrigues (1840) and streamlined by Gibbs, who defined the eponymous vectors parameterizing rotation axes and angles,
$$
\vec b = \hat x ~\tan \alpha/2, \qquad \vec a = \hat z ~\tan \beta/2 ,
$$
whose dot product vanishes in your simplified case, so the combined rotation formula of Gibbs reduces to just
$$
\vec a + \vec b + \vec a \times \vec b = \hat x ~\tan \alpha/2 + \hat z ~\tan \beta/2 + \hat y ~\tan \alpha/2 ~\tan \beta/2.
$$
(Check with limiting cases, such as right angles. So $\alpha=\beta=\pi/2$ leaves the vector (1,1,1)T invariant, so the axis of the combined rotation, above.)
This is to be composed with $$ -\vec a - \vec b + \vec a \times \vec b , $$ which I will not pursue: the Gibbs denominator is nontrivial, now, and the inversions messy, unless you may simplify them by symmetry. The Gibbs vector you'd obtain in the end would specify the axis direction of the group commutator effective rotation, and the half angle of the effective rotation involved.
PS Modern users prefer to use the simple doublet representation (Pauli matrices) which motivates the half-angles and can be more easily checkable. It is, of course, entirely equivalent: these are all faithful irrep group identities, so the answers do not depend on the representation. Evidently, then, $R_x(\alpha)R_z(\beta)=R_{\hat k}(2\arccos(\cos \alpha/2 ~\cos \beta/2))$ , where $\hat k= (\hat x \sin \alpha/2 ~\cos \beta/2+\hat z \cos \alpha/2 ~\sin \beta/2+\hat y \sin \alpha/2 ~\sin \beta/2)/\sin(\arccos (\cos \alpha/2 ~\cos \beta/2))$ .
PS PS Some textbooks, including good ones, sometimes insert rotation matrix combinations into Lie algebra commutators, thereby going outside the group, in sharp contrast to group commutators which stay in the group. While such entities, illustrated with right-angle rotations of books and such, can be illustrative of noncommutativity, they nevertheless may generate confusions put to rest here.
