What is the overlap between different spin component eigenstates?

Question

I am trying to find an expression for the overlap between the eigenstates of different spin component operators in a spin-S system. Say I have operators $\hat{S}_i,~i=x,y,z$ with eigenvalue equations $\hat{S}_i|S,m_i\rangle=m_i|S,m_i\rangle$. I now want to find an expression for $\langle S,m_i|S,m_j\rangle,~i\neq j$.

My intuition on this is that $$|S,m_i\rangle=\sum_{m_j=-S}^S c_{m_im_j}|S,m_j\rangle,$$ based on the fact that we know from the familiar case of the spin-1/2 system that $$|1/2,m_x=\pm1/2\rangle=\frac{1}{\sqrt{2}}\left(|1/2,m_z=+1/2\rangle\pm|1/2,m_z=-1/2\rangle\right)$$ By extension, there should be some formula for the coefficients $c_{m_i m_j}$ for a general spin-S system.

This feels like it should be a standard result (akin to the tabulated Clebsch-Gordan coefficients for transforming between the spin coupled and uncoupled representation bases), but I can't find a good reference anywhere. I have tagged this as group-theory and representation-theory as well, since I suspect that the solution is probably to be found in Lie theory, in the representations of SU(2).

Answer 1

You are essentially trying to change basis from eigenvectors of $\hat S_z$ to eigenvectors of $\hat S_x$, for example, for an arbitrary fixed S. (The other two cases are similar, and you may trivially carry them out yourself, as suggested here.) Feynman, vIII does your S = 1/2 case. The short answer is that your quantities are Wigner rotation d matrices for an angle π/2. Indeed, for arbitrary angles, these matrices are displayed next to the C-G coefficients in tabulations, such as the PDG.

First for the answers, before explanations. To find your spin 1/2 answer, you dotted $$\langle 1/2, m_z=1/2|\leadsto (1,0), \qquad \langle 1/2, m_z=-1/2|\leadsto (0,1),$$ onto the eigenvectors of $\hat S_x$, $$ |1/2,m_x=1/2\rangle \leadsto \begin{pmatrix} 1\\1\end{pmatrix} /\sqrt{2}, \qquad |1/2,m_x=-1/2\rangle \leadsto \begin{pmatrix} -1\\1\end{pmatrix} /\sqrt{2}. $$ You then got the matrix of your basis change, up to a phase convention, $$ \langle 1/2, m_z\!\!=\!\!1/2|1/2,m_x\!\!=\!\!1/2\rangle\!\! = 1/\sqrt{2}, \quad \langle 1/2, m_z\!\!=\!\!1/2|1/2,m_x\!\!=\!\!-1/2\rangle \!\!= - 1/\sqrt{2}\\ \langle 1/2, m_z\!\!=\!\!-1/2|1/2,m_x\!\!=\!\!1/2\rangle \!\!= 1/\sqrt{2}, \quad \langle 1/2, m_z\!\!=\!\!-1/2|1/2,m_x\!\!=\!\!-1/2\rangle \!\!= 1/\sqrt{2}. $$ This orthogonal matrix associates eigenvectors of $\hat S_x$ to eigenvectors of $\hat S_z$, so $$|m_x\rangle= \sum_{m_z} |m_z\rangle \langle m_z|m_x\rangle.$$

For higher spins, you just dot the eigenvectors provided. For instance, $$ \langle 1, m_z\!\!=\!\!1~|~1,m_x\!\!=\!\!1\rangle =1/2=\langle 1, m_z\!\!=\!\!-1|1,m_x\!\!=\!\!1\rangle \\ \langle 1, m_z\!\!=\!\!0|1,m_x\!\!=\!\!1\rangle =1/\sqrt{2} , \\ \langle 3/2, m_z\!\!=\!\!3/2 ~|~ 3/2,m_x\!\!=\!\!3/2\rangle = {1\over 2\sqrt{2}}~,\\ \langle 2, m_z\!\!=\!\!2 ~|~ 2,m_x\!\!=\!\!2\rangle = {1\over 4 }~ ... $$

But ... do you need to be finding the eigenvectors of the spin operators for arbitrary spins, by hand, all the time? Not really. Observe $$\hat S_z=e^{i{\pi/2} \hat S_y} \hat S_x e^{-i{\pi/2} \hat S_y} ,$$ that is, a right angle rotation around y rotates $\hat S_x$ to $\hat S_z$, and so diagonalizes it. It is straightforward to confirm the spin 1/2 case with standard Pauli matrix exponentiation to catch your signs, $$ e^{i{\pi/4} \sigma_y} \sigma_x e^{-i{\pi/4} \sigma_y} = \sigma_z ~. $$

Observe the right exponential is a rotation matrix transforming an eigenvector of $\hat S_z$ to an eigenvector of $\hat S_x$ with the same eigenvalue, just like the transformation matrix sought. So the eigenvectors and hence basis change coefficients are residing in the columns of the rotation Wigner d-matrices in the right angle limit. (It would be worth your while to confirm/check a few cases; further, in Feynman's (18.35) in that limit.)