What is a rotation group and how do we get its unitary representation? The rotation group is ${\rm SO(3)}$. It is the group of $3\times 3$ orthogonal matrices $\{g(\theta)\}$ with unit determinant. So these are already defined in terms of $3\times 3$ matrices. But we use unitary representation $\{U(g(\theta))\}$ of the rotation group in quantum mechanics. What does that even mean?
How do we define rotation group if not in terms of explicit ${\rm SO(3)}$ matrices? Is this already not a representation? Isn't the definition of rotation group already in terms of this representation?
Given the elements of the rotation group (i.e., the ${\rm SO(3)}$ matrices $g(\theta)$) how do we get, $U(g(\theta))$?
 A: The point here is that, if $\omega\in SO(3)$, and if $\omega_1\cdot \omega_2=\omega \in SO(3)$ is the combination rule on abstract elements, then a representation (by matrices) $U$ is a map $\omega\mapsto U(\omega)$ so that the rule
$$
\omega_1\cdot \omega_2=\omega  \quad \Rightarrow \quad U(\omega_1)\cdot U(\omega_2)=U(\omega) \tag{1}
$$
for any $\omega_1,\omega_2,\omega\in SO(3)$ is also satisfied by the matrices $U(\omega)$ representing the elements.  There is a theorem stating that, for $SO(3)$ and a bunch of others, all representation are equivalent to unitary representations, so that $U(\omega^{-1})=U^{-1}(\omega)=U^\dagger(\omega)$.  
Although the so-called defining representation is in terms of a $3$-dimensional space on which $SO(3)$ acts "naturally", there may be matrices of dimension other than $3$ that satisfy the basic composition law of $\omega_1\cdot \omega_2=\omega $ or its
matrix version of Eq.(1).
You can "obtain" a representation by larger matrices by tensoring and decomposing the resulting representation.  For instance, if 
$\{\vert {1}\rangle ,\vert {2}\rangle,\vert {3}\rangle\}$ are a basis for the $3$-dimensional irrep of $SO(3)$, then the set$\{\vert i\rangle\vert j\rangle\}$ spans a 9-dimensional space with
$$
U(\omega)\left[\vert i\rangle\otimes\vert j\rangle\right]:=
\left[U(\omega)\vert i\rangle\right]\otimes \left[U(\omega)\vert j\rangle\right]
$$
will provide you with a $9$-dimensional representation, which turns out to be reducible.  Note that I'm abusing the notation here because on the left I have $U$ as a $9\times 9$ matrix but on the right the $U$'s are $3\times 3$ matrices. 
 In fact, the $9\times 9$ representation is reducible: it contains $L=2,1,0$, i.e. irreducible pieces of dimensions $5,3$ and $1$.  The $L=2$ and $L=0$ irreps are spanned by symmetric combinations like $\vert 1\rangle\vert 2\rangle+ \vert 2\rangle\vert 1\rangle$ etc, while the $L=1$ contains antisymmetric pieces.  
In cases other than $SO(3)$ (or $SU(2)$), one can also obtain inequivalent representations by taking the conjugate.  The simplest example would be $SU(3)$, where the defining representation ($3\times 3$) is often denoted by $\textbf{3}$ or $(1,0)$ in the Dynkin scheme, and where its (non-equivalent) conjugate is denoted by $\textbf{3*}$ or $(0,1)$.   One can then construct any representation by tensoring a suitable number of copies of $(1,0)$ and $(0,1)$ and decomposing the result.  
Note that $SO(3)$ representations  (and also $SU(2)$ representations) are "self-conjugate" in the sense that taking the conjugate yields the same representation.
A: [about SO(3)] 
SO(3) is an abstract group with a lot of well-known properties (Lie, compact, topological etc). The representation of SO(3) via 3d, orthogonal, real-valued maricies is one of many possible ones. But this is, by definition, a faithful representation, i.e. every group member has a distinct matrix that corresponds to it.
[about getting unitary reps]
The rotation matricies are orthogonal and real-valued, so they are already unitary, thus, technically, the question is moot.
If your question is how to go from the general orthogonal, 3d, real-valued matrix with unity determinant to its representation as $R=\exp\left(\dots\right)$, i would suggest diagonalizing the matrix. 
You will find that this does not work over the space $\mathbb{R}^3$, but it does work over $\mathbb{C}^3$. Quite simple reasoning can show that any
$R\in \mathbb{R}^3\times\mathbb{R}^3$ with $\det R=1$ and $R^T=R^{-1}$ can be diagonalized in the complex space with eigenvalues:
$\lambda_{1,2,3}=\exp\left(\pm i\phi\right), 1$ for some $\phi\in\mathbb{R}$
Thus $R=V \exp\left(i\left(\begin{array}\\
\phi &&0 && 0\\
0 && -\phi && 0\\
0 && 0 && 0
\end{array}\right)\right) V^{\dagger}$
Where the $V$ is the unitary matrix with eigenvectors. Now simply take the $V$ matricies into the exponential and you will have your representation.
