What is the 4-dimensional matrix representation of rotation operator? The rotation operator is $$\exp\left(-i\frac{\theta}{2}\boldsymbol{J}\cdot\hat{\boldsymbol{n}}\right).$$ 


*

*If $\boldsymbol{\sigma}$ is the Pauli matrix, the operator can be written as a matrix form $$\boldsymbol{1}\cos(\phi/2)-i\boldsymbol{\sigma}\cdot\hat{\boldsymbol{n}}\sin(\phi/2).$$ 

*But when $J$ is the spin-3/2 operator, $J$ is 4-dimensional. Is there a matrix representation of operator $\exp\left(-i\frac{\theta}{2}\boldsymbol{J}\cdot\hat{\boldsymbol{n}}\right)$? I find that when $\{J_x,J_y\}\neq0$ for spin-3/2, not like Pauli matrices. 

*What is the case when $J$ is spin-1 operator?
 A: There is a general expression in my article A Compact Formula for Rotations
as Spin Matrix Polynomials, SIGMA 10 (2014), 084, to the effect that, e.g., for the doublet,
\begin{gather*}
e^{i(\theta/2)(\hat{\boldsymbol{n}}\cdot\boldsymbol{\sigma})}=I_{2}\cos{\theta/2}+i(\hat{\boldsymbol{n}}\cdot\boldsymbol{\sigma})\sin{\theta/2},
\end{gather*}
and the triplet, $j=1$, so $J_{3}=\mathrm{diag}(1,0,-1)$,
\begin{gather*}
e^{i\theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})}=I_{3}+i(\boldsymbol{\hat
{n}}\cdot\boldsymbol{J})\sin{\theta}+(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})^{2}(\cos\theta-1)
\\
\phantom{e^{i\theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})}}
=I_{3}+(2i\hat{\boldsymbol{n}}\cdot\boldsymbol{J}\sin(\theta/2))\cos
(\theta/2)+\tfrac{1}{2}(2i\hat{\boldsymbol{n}}\cdot\boldsymbol{J}\sin (\theta/2))^{2}.
\end{gather*}
For the quartet, $j=3/2$,
\begin{gather}
e^{i \theta (\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} = I_4 \cos (\theta/2)\left(1+\tfrac{1}{2}\sin^2
(\theta/2)\right)+(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin (\theta/2))\left(1+\tfrac{1}{6} \sin^2 (\theta/2)
\right)
\nonumber
\\
\phantom{e^{i \theta (\hat{\boldsymbol{n}}\cdot\boldsymbol{J})}=}{}
+\frac{1}{2!} \bigl (2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J}\sin (\theta/2) \bigr)^2 \cos (\theta/2)+\frac {1}{3!}
\bigl (2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J}\sin (\theta/2) \bigr)^3.
\label{quartet}
\end{gather}
For the quintet, $j=2$,
\begin{gather*}
e^{i \theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} = I_5+(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin(\theta/2))
\cos(\theta/2)\left(1+\tfrac{2}{3}\sin^2(\theta/2)\right)
\\
\phantom{e^{i \theta (\hat{\boldsymbol{n}}\cdot\boldsymbol{J})}=}{}
+\frac{1}{2!} {(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin(\theta/2))^2}\left(1+\tfrac{1}{3} \sin^2 (\theta/2)\right)
\\
\phantom{e^{i \theta (\hat{\boldsymbol{n}}\cdot\boldsymbol{J})}=}{}
+\frac{1}{3!} {(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin(\theta/2))^3} \cos(\theta /2) +\frac{1}{4!} (2i
\hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin (\theta/2))^4.
\end{gather*}
For the sextet, $j=5/2$,
\begin{gather*}
e^{i \theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} = I_6 \cos(\theta/2)\left(1+ \tfrac{1}{2} \sin^2
(\theta/2+\tfrac{3}{8} \sin^4 (\theta/2)\right)
\\
\phantom{e^{i \theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} =}{}
+(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin(\theta /2))\left(1+\tfrac{1}{6}\sin^2(\theta/2)
+\tfrac{3}{40}\sin^4(\theta /2)\right)
\\
\phantom{e^{i \theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} =}{}
+\frac{1}{2!} {(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin(\theta/2))^2} \cos(\theta /2)
\left(1+\tfrac{5}{6}\sin^2(\theta/2)\right)
\\
\phantom{e^{i \theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} =}{}
+\frac{1}{3!} {(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin (\theta/2))^3}\left(1+\tfrac{1}{2}\sin^2(\theta/2)
\right)
\\
\phantom{e^{i \theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} =}{}
+\frac{1}{4!} {(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin(\theta/2))^4}\cos(\theta /2) +\frac{1}{5!} {(2i
\hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin (\theta/2))^5}.
\end{gather*}
etc...
There is a simple pattern and compact formula for arbitrary spin detailed in that paper.
A: One way to construct the explicit representations of rotations ($\mathrm{SO}(3)$) is to start from the matrix representation of the raising and lowering operators in the $J_z$ basis, 
$$J_\pm|j\,m\rangle = \hbar \sqrt{j(j+1) - m(m\pm 1)}|j\, (m\pm1)\rangle,$$
(note that when written as matrices $J_\pm$ have elements above/below the main diagonal, depending on how you order the $J_z$ eigenstates). Then invert the definition of $J_\pm$, $J_{\pm} \equiv J_x \pm i J_y$, to get:
$$\begin{align}
J_x &= \frac{J_+ + J_-}{2},\ \mathrm{and} \\
J_y &= \frac{J_+ - J_-}{2i}.
\end{align}$$
Recall that in the $J_z$ basis, $J_z$ is diagonal with matrix elements:
$$\langle j'\, m' | J_z | j\, m\rangle = \delta_{j\, j'} \delta_{m\, m'} \hbar m.$$
You can then insert the explicit matrices for $J_x$, $J_y$, and $J_z$ into $\exp\left(-i \frac{\theta}{\hbar} \hat{n} \cdot \vec{J}\right)$ to construct a specific matrix (Note that the Pauli matrices are given by $\sigma_i = \frac{2}{\hbar} J_i$ for the spin 1/2 representation). To produce a general form for the matrices is a little tricky, since you'll have to examine the Taylor expansion, construct a set of linearly independent basis matrices from it, group them, and identify the trig functions multiplying each of the basis matrices.
If a numerical solution will suffice, programming languages like Julia and MATLAB have a function called expm (for "exponential of a matrix").
If what you're interested is the $j=1$ representation, that's extremely well studied, with multiple answers available for the taking. I would suggest comparing those answers (especially Rodrigues's rotation formula) with what you construct using the above with $j=1$ to practice before moving on to $j = 3/2$ or higher.
A: There are several ways to proceed.  
First, be aware that Mathematica has a built-in function called WignerD
and this function will give you the matrix element of a rotation matrix.  I have found that this function does not seem to use the same parametrization as Varshalovich, Dmitriĭ Aleksandrovich, Anatolij Nikolaevič Moskalev, and Valerii Kel'manovich Khersonskii. Quantum theory of angular momentum. 1988., which in my opinion remains the bible.  It seems you need to use the negative of all the angles in WignerD  to obtain the formulas of Varshalovich.
There are various closed form expressions for 
\begin{align}
d^j_{mm'}(\beta)&:= \langle jm\vert R_y(\beta)\vert jm'\rangle \, 
\end{align}
such as
\begin{align}
d^j_{mm'}(\beta)&=(-1)^{j-m'} 
\sqrt{(j+m)!(j-m)!(j+m')!(j-m')!}\\
&\times \sum_k (-1)^k
\frac{\left(\cos\frac{1}{2}\beta\right)^{m+m'+2k}
\left(\sin\frac{1}{2}\beta\right)^{2j-m-m'-2k}}
{k!(j-m-k)!(j-m'-k)!(m+m'+k)!}\, .
\end{align}
There is also an expression in terms of Jacobi polynomials:
\begin{align}
d^j_{mm'}(\beta)&=\xi_{mm'} 
\sqrt{\frac{s!(s+\mu+\nu)!}{(s+\mu)!(s+\nu)!}}
\left(\sin\textstyle\frac{1}{2}\beta\right)^\mu
\left(\cos\textstyle\frac{1}{2}\beta\right)^\nu
P_s^{\mu,\nu}(\cos\beta)
\end{align}
with 
$$
\mu=\vert m-m'\vert\, ,\quad
\nu=\vert m+m'\vert\, ,\quad
s=j-\frac{1}{2}(\mu+\nu)
$$
and the phase 
$$
\xi_{mm'}=\left\{\begin{array}{cc}
1&\quad \hbox{if } m'\ge m\, \\
(-1)^{m'-m}&\quad \hbox{if } m'<m\, .\end{array}\right.
$$
Finally, there is a method based on recursion relations.  This is described in details in Wolters, G. F. "Simple method for the explicit calculation of d-functions." Nuclear Physics B 18.2 (1970): 625-653. Starting with 
$$
\langle jm\vert R_y(\beta) L_x\vert jm'\rangle
$$
one can obtain a recursion relation 
\begin{align}
&\sqrt{(j-m')(j+m'+1)}d^j_{m,m'+1}(\beta)
+\sqrt{(j+m')(j-m'+1)}d^j_{m,m'-1}(\beta)\\
&\qquad = 2\hbox{cosec}(\beta)(m'\cos\beta-m)d^j_{mm'}(\beta)
\end{align}
The function
\begin{align}
d^j_{mj}(\beta)&={2j \choose j+m}^{1/2}
\left(\cos\textstyle\frac{1}{2}\beta\right)^{j+m}
\left(\sin\textstyle\frac{1}{2}\beta\right)^{j-m}\, ,\\
\end{align}
can be used as a seed for the recursion.
As a matrix with elements $d^{3/2}_{mm'}(\beta)$, the explicit results for $j=3/2$ is
\begin{align}
&R_y(\beta)=\\
&{\scriptsize\left(
\begin{array}{cccc}
 \cos ^3\left(\frac{\beta }{2}\right) & -\sqrt{3} \cos ^2\left(\frac{\beta }{2}\right) \sin \left(\frac{\beta }{2}\right) & \sqrt{3} \cos \left(\frac{\beta
   }{2}\right) \sin ^2\left(\frac{\beta }{2}\right) & -\sin ^3\left(\frac{\beta }{2}\right) \\
 \sqrt{3} \cos ^2\left(\frac{\beta }{2}\right) \sin \left(\frac{\beta }{2}\right) & \frac{1}{2} \cos \left(\frac{\beta }{2}\right) (3 \cos (\beta )-1) &
   -\frac{1}{2} (3 \cos (\beta )+1) \sin \left(\frac{\beta }{2}\right) & \sqrt{3} \cos \left(\frac{\beta }{2}\right) \sin ^2\left(\frac{\beta }{2}\right) \\
 \sqrt{3} \cos \left(\frac{\beta }{2}\right) \sin ^2\left(\frac{\beta }{2}\right) & \frac{1}{2} (3 \cos (\beta )+1) \sin \left(\frac{\beta }{2}\right) &
   \frac{1}{2} \cos \left(\frac{\beta }{2}\right) (3 \cos (\beta )-1) & -\sqrt{3} \cos ^2\left(\frac{\beta }{2}\right) \sin \left(\frac{\beta }{2}\right) \\
 \sin ^3\left(\frac{\beta }{2}\right) & \sqrt{3} \cos \left(\frac{\beta }{2}\right) \sin ^2\left(\frac{\beta }{2}\right) & \sqrt{3} \cos ^2\left(\frac{\beta
   }{2}\right) \sin \left(\frac{\beta }{2}\right) & \cos ^3\left(\frac{\beta }{2}\right) \\
\end{array}
\right)}
\end{align}
with the columns and rows ordered as $3/2,1/2,-1/2,-3/2$.
