What is the overlap between different spin component eigenstates? I am trying to find an expression for the overlap between the eigenstates of different spin component operators in a spin-S system. Say I have operators $\hat{S}_i,~i=x,y,z$ with eigenvalue equations $\hat{S}_i|S,m_i\rangle=m_i|S,m_i\rangle$. I now want to find an expression for $\langle S,m_i|S,m_j\rangle,~i\neq j$.
My intuition on this is that
$$|S,m_i\rangle=\sum_{m_j=-S}^S c_{m_im_j}|S,m_j\rangle,$$
based on the fact that we know from the familiar case of the spin-1/2 system that $$|1/2,m_x=\pm1/2\rangle=\frac{1}{\sqrt{2}}\left(|1/2,m_z=+1/2\rangle\pm|1/2,m_z=-1/2\rangle\right)$$
By extension, there should be some formula for the coefficients $c_{m_i m_j}$ for a general spin-S system.
This feels like it should be a standard result (akin to the tabulated Clebsch-Gordan coefficients for transforming between the spin coupled and uncoupled representation bases), but I can't find a good reference anywhere. I have tagged this as group-theory and representation-theory as well, since I suspect that the solution is probably to be found in Lie theory, in the representations of SU(2).
 A: You are essentially trying to change basis from eigenvectors of $\hat S_z$ to eigenvectors of $\hat S_x$, for example, for an arbitrary fixed S. (The other two cases are similar, and you may trivially carry them out yourself, as suggested here.) Feynman, vIII does  your S = 1/2 case. The short answer is that your quantities are Wigner rotation d matrices for an angle π/2. Indeed, for arbitrary angles, these matrices are displayed next to the C-G coefficients in tabulations, such as the PDG.
First for the answers, before explanations. To find your spin 1/2 answer, you dotted
$$\langle 1/2, m_z=1/2|\leadsto  (1,0), \qquad \langle 1/2, m_z=-1/2|\leadsto  (0,1),$$
onto the eigenvectors of  $\hat S_x$,
$$
|1/2,m_x=1/2\rangle \leadsto \begin{pmatrix} 1\\1\end{pmatrix}   /\sqrt{2}, \qquad |1/2,m_x=-1/2\rangle \leadsto \begin{pmatrix} -1\\1\end{pmatrix}    /\sqrt{2}.
$$
You then got the  matrix of your basis change, up to a phase convention,
$$
\langle 1/2, m_z\!\!=\!\!1/2|1/2,m_x\!\!=\!\!1/2\rangle\!\! =  1/\sqrt{2},  \quad \langle 1/2, m_z\!\!=\!\!1/2|1/2,m_x\!\!=\!\!-1/2\rangle \!\!= - 1/\sqrt{2}\\
\langle 1/2, m_z\!\!=\!\!-1/2|1/2,m_x\!\!=\!\!1/2\rangle \!\!=  1/\sqrt{2},  \quad \langle 1/2, m_z\!\!=\!\!-1/2|1/2,m_x\!\!=\!\!-1/2\rangle \!\!= 1/\sqrt{2}.
$$
This orthogonal matrix associates eigenvectors of  $\hat S_x$ to eigenvectors of $\hat S_z$, so $$|m_x\rangle= \sum_{m_z} |m_z\rangle \langle m_z|m_x\rangle.$$
For higher spins, you just dot the eigenvectors provided. For instance,
$$
\langle 1, m_z\!\!=\!\!1~|~1,m_x\!\!=\!\!1\rangle =1/2=\langle 1, m_z\!\!=\!\!-1|1,m_x\!\!=\!\!1\rangle \\   \langle 1, m_z\!\!=\!\!0|1,m_x\!\!=\!\!1\rangle =1/\sqrt{2} , \\
\langle 3/2, m_z\!\!=\!\!3/2 ~|~ 3/2,m_x\!\!=\!\!3/2\rangle = {1\over 2\sqrt{2}}~,\\
\langle 2, m_z\!\!=\!\!2 ~|~ 
2,m_x\!\!=\!\!2\rangle = {1\over 4 }~ ...
$$
But ... do you need to be finding the eigenvectors of the spin operators for arbitrary spins, by hand, all the time? Not really. Observe $$\hat S_z=e^{i{\pi/2} \hat S_y} \hat S_x e^{-i{\pi/2} \hat S_y} ,$$ that is, a right angle rotation around y rotates  $\hat S_x$ to $\hat S_z$, and so diagonalizes it. It is straightforward to confirm the spin 1/2 case with standard Pauli matrix exponentiation to catch your signs,
$$
 e^{i{\pi/4} \sigma_y} \sigma_x e^{-i{\pi/4} \sigma_y} = \sigma_z ~.
$$
Observe the right exponential is a rotation matrix transforming an eigenvector of $\hat S_z$ to an eigenvector of $\hat S_x$ with the same eigenvalue, just like the transformation matrix sought.  So the eigenvectors and hence basis change coefficients are residing in the columns of the rotation Wigner d-matrices in the right angle limit. (It would be worth your while to confirm/check a few cases; further,  in Feynman's (18.35) in that limit.)
