Matrix representation of rotation operators in QM The usual matrix form of an operator $X$ is given by matrix components $$\langle a''| X | a' \rangle $$ where $|a' \rangle$ forms a basis for the ket space. In the case where we define matrix of rotation (where $|j,m \rangle$ are eigenstates of $J^2$ and $J_z$) $$\mathcal{D}_{m',m}^{(j)}(R) = \langle j, m' |\text{exp} \bigg(\frac{-i \mathbf{J} \cdot \hat{n} \phi}{\hbar} \bigg) |j,m \rangle$$ we go through all of $j$'s and $m$'s and get a block matrix representation of the rotation $\text{exp}\bigg(\frac{-i \mathbf{J} \cdot \hat{n} \phi}{\hbar} \bigg)$.
That fits with the above definition of matrix representation of a rotation operator. But in a text (Sakurai's 'Modern Quantum Mechanics' pages 196-197) it states that for specific $j$, we still get a matrix representation of a rotation operator, would this not imply that we are not using the full set of the basis $|j,m \rangle$ for this representation, how is that a valid matrix representation of an operator?
 A: I'll try to help here, with some details I believe to be of aid in this question. If there's anything in the need of fixing here, please let me know. 
First of all, let's introduce some terminology. Given a group $G$ and a vector space $V$, a representation of $G$ on the vector space $V$ is a homomorphism $\rho : G\to GL(V)$. We then say that $V$ carries a representation of $G$.
If $V$ is a complex inner product space, and $\rho(g)$ is unitary for each $g\in G$, then $\rho$ is said a unitary representation.
It turns out the meaning of this is that the elements of $G$ act on the vector space $V$.
There are lots of constructions that can be acrried out with representations. One of them is the so-called direct sum of representations, which as can be expected is tied to the direct sum of vector spaces.
The point is that considering the group $G$ fixed if you have a representation $(\rho,V)$ and another one $(\sigma,W)$, then you can induce a representation $(\rho\oplus \sigma, V\oplus W)$ on the vector space $V\oplus W$ by setting 
$$\rho\oplus\sigma(g)(v,w)=(\rho(g)v, \sigma(g)w).$$
It turns out direct sums allows you to decompose representations into smaller components that build up the representation you started with.
For that matter one defines a subrepresentation of $(\rho,V)$ to be given by a subspace $W$ of $V$ invariant under the $\rho$-action of $G$. It turns out then that $(\rho,W)$ is a representation itself. With this terminlogy we say that a representation is irreducible if it doesn't have any proper subrepresentations, i.e., if it doesn't have "smaller parts".
Now, let's talk about rotations. The rotation group in three-dimensions is $SO(3)$. Quantum Mechanics describes systems by Hilbert spaces, which are complex complete inner product spaces. Transformations should be implemented by unitary transformations, to keep the physical quantities intact.
Thus if we want to know how rotations affect the states of the systems we are studying, we need to consider the action of $SO(3)$  on the Hilbert space $\mathcal{H}$, requiring unitarity, that is, we must consider unitary representations of $SO(3)$.
This is what your $\mathcal{D}(R)$ is. The issue is that you want the irreducible representations of $SO(3)$. This can be tackled by studying its Lie Algebra, which is the angular momentum algebra. The result one finds is: (i) there are no infinite-dimensional unitary irreducible representations of $SO(3)$, and (ii) there is one finite-dimensional representation of $SO(3)$ for each integer $j$ with dimensionality $2j+1$. These representations are denoted $D^{j}$.
But these are just the irreducibles. You can build up representations with the direct sum. When you take the direct sum of two such representations, the new one will be given by a block-diagonal matrix.
So when you let $j$ run through all integer values for example, you are actually considering the representation given by taking the direct sum over all these values. When you fix one $j$, you are considering just one particular irreducible representation acting on a subspace of state space.
For more informations, search for the irreducible representations of $SO(3)$, and you'll probably find the whole construction rigorously made.
A: I hope I understand you right.  
In general the game is to take the infinite-dimensional Hilbert space and break it into a sum of blocks.  This partitioning of course is not unique but quite clearly there is advantage in choosing a partitioning where the operators mix the blocks as little as possible.  In particular, if your operator does not mix the blocks and the blocks are finite dimensional, then by linearity you can work in each finite-dimensional block, an obvious advantage over working in an infinite-dimensional space.  (This is clearly the optimal situation, not always realized in practice.)
In the case of rotations, the partitioning will involve basis states of the form $\vert \alpha j m\rangle$ where $\alpha$ is any other quantum number that is not $j$ or $m$.  In this basis your Hilbert space now looks (cartoon version) like 
$$
\left(\begin{array}{c|c|c|c|c}
\ldots&\hbox{$
\begin{array}{c}
\phantom{\hbox{a}}\\
(2j_1+1)\times (2j_1+1)\\
\phantom{\hbox{a}}\end{array}$}& 0 & 0&\ldots \\
\hline
\ldots&0&\hbox{$
\begin{array}{c}
\phantom{\hbox{a}}\\
(2j_2+1)\times (2j_2+1)\\
\phantom{\hbox{a}}\end{array}$}& 0&\ldots \\ \hline 
\vdots & \vdots & 0 &
\hbox{$
\begin{array}{c}
\phantom{\hbox{a}}\\
\ddots\\
\phantom{\hbox{a}}\end{array}$}
\end{array}\right)
$$
with the blocks spanned by states $\vert\alpha \ell m\rangle $, one pair $(\alpha,\ell)$ for each block.
In this basis, the rotation operators connects states
not only with the same $j$ but also with the same $\alpha$, i.e. 
$\vert\alpha jm\rangle$ and $\alpha j m'\rangle$: even if there are multiple states with the same $j$ value (this could occur for instance in the hydrogen atom or the 3d oscillator, where some specific $\ell$ values will occur for different energies), the rotation operator will not connect those, i.e. 
$$
\langle \beta j m\vert R\vert \alpha jm
\rangle =\delta_{\alpha\beta}D^{j}_{mm'}(R)\, .
$$
Thus, if I understand your question well, the statement about getting a matrix representation of the rotation operator for a specific $j$ really means that the rotation operators acting in an infinite dimensional space can be broken up in $(2j+1)\times (2j+1)$ pieces (obviously there is more than one possible value of $j$), and that inside each $j$ subspace you have a representation of the rotation operator.
Your kets $\vert a\rangle$ and $\vert a'\rangle$ will live in the infinite-dimensional Hilbert space.  Since by assumption $\vert \alpha \ell m\rangle$ form a complete set of orthonormal states in this infinite-dimensional space, one should first decompose each
state as
\begin{align}
\vert a\rangle&=\sum_{\alpha \ell m}\vert \alpha \ell m\rangle\langle 
\alpha \ell m\vert a\rangle \, ,\\ 
\vert a'\rangle&=\sum_{\beta \ell' m'}\vert \beta \ell' m'\rangle\langle 
\beta \ell' m'\vert a'\rangle\, ,
\end{align}
so that
\begin{align}
\langle a'\vert R \vert a\rangle&= 
\sum_{\alpha\beta \ell\ell' m m'}
\langle a'\vert \beta \ell' m'\rangle\langle 
\beta \ell' m'
R\vert \alpha \ell m\rangle\langle 
\alpha \ell m\vert a\rangle\, ,\\
&=\sum_{\alpha\beta \ell\ell' m m'}
\langle a'\vert \beta \ell' m'\rangle\langle 
\beta \ell' m'
R\vert \alpha \ell m\rangle\langle 
\alpha \ell m\vert a\rangle\delta_{\alpha\beta}\delta_{\ell\ell'}\, ,\\
&=\sum_{\alpha\ell m m'}
\langle a'\vert \alpha \ell m'\rangle
D^\ell_{m'm}(R)
\langle \alpha \ell m\vert a\rangle
\end{align}
so that the rotation operator operates within a subblock $\alpha\ell$, but the there is a sum of such subblocks. 
As a specific example, consider the states of a 3d harmonic oscillator. The only quantum number needed in addition to $\ell$ and $m$ is $n$ so $\alpha=n$. 
Labelling states as $\vert n\ell m$, we have (for instance), the state
$\vert 000\rangle$, $\vert 11M$, with $M=-1,0,1$, $\vert 200\rangle$, $\vert 22m\rangle$ with $m=-2,\ldots,2$ etc.  The list goes on since the Hilbert space is infinite-dimensional.
The Hilbert space will contain the following blocks:


*

*one $1\times 1$ block spanned by $\vert 000\rangle$,

*one $1\times 1$ block spanned by $\vert 200\rangle$,

*one $3\times 3$ block spanned by $\{\vert 11M\rangle, M=-1,0,1\}$,

*one $5\times 5$ block spanned by $\{\vert 22m\rangle, m=-2, \ldots, 2\}$,

*etc.


Thus, if you have the general states 
\begin{align}
\vert a\rangle = \sum_{n\ell m} b_{n\ell m} \vert n\ell m\rangle\, ,
\qquad 
\vert a'\rangle = \sum_{n\ell m} c_{n'\ell' m'} \vert n'\ell' m'\rangle\, ,
\end{align}
then
\begin{align}
\langle a'\vert R\vert a\rangle&=\sum_{n\ell mm'}
b_{n\ell m}c^*_{n\ell m'} D^\ell_{m'm}(R)\, .
\end{align}
