The symmetry group and representation of spin-$N$ particle I am confused with the symmetry group and the representation of spin-$N$ particles, and will appreciate any help or suggestions of reference.
There are $2N+1$ internal states for a (massive) spin-$N$ particle. These internal states define a $2N+1$-dimensional Hilbert space. It seems to be resonable that, the associate symmetry group is $SU(2N+1)$.
However, it seems also possible that, the $2N+1$ states correspond to the $2N+1$-dimension irreducible representation of $SU(2)$.
Are there some relations between the above two situations, or I just confuse some basic concepts?
 A: There are a few confusions here, so I'll make my answer modular.


*

*A representation $R$ of a group $G$ is a vector space $V$ along with operators $R(g)$ for $g \in G$ that act on that space. Thus, given just a vector space, it's meaningless to say what it's a representation of; you need to specify the operators. 

*Given just a vector space $V$, it is meaningless to ask what $G$ is. Any vector space can be regarded as a representation of literally any group $G$ whatsoever, with $R(g)$ simply being, e.g. the identity matrix for every $g \in G$.

*In physics, we begin with a vector space $\mathcal{H}$, the state space of a quantum system. Then we identify physical operations, such as rotations, isospin rotations, color rotations, and so on. Each of these operations, as a set, has the structure of a group ($SU(2)$, $SU(2)$, and $SU(3)$). 

*We postulate that these operations (e.g. the physical operation of rotating a system) are associated with operators on $\mathcal{H}$, making $\mathcal{H}$ into a representation of the corresponding group.

*We say the system has, e.g. rotational symmetry if the Hamiltonian commutes with all of the rotation operators. This implies that the states in every irreducible representation have the exact same energy, which is essentially what makes symmetry analysis useful.


Okay, now let's get to your actual question. We're given the $2N+1$ states of a spin $N$ particle, and you want to know if it's symmetric under $SU(2N+1)$.


*

*The states indeed form a representation of $SU(2)$, where the $SU(2)$ comes from rotations; that's just by the definition of a spin $N$ particle. If we have rotational symmetry, then the states must all have the same energy.

*Note that spin is almost exclusively used to describe representations of the rotation operators. There is no spin associated with a representation of $SU(3)$. Even the two-state representation of isospin, which also has symmetry group $SU(2)$, is not called 'spin 1/2' or even 'isospin 1/2'. Instead it's usually called an 'isospin doublet' or a '$2$ of isospin'.

*Mathematically, the states indeed can form a representation of $SU(2N+1)$, where the representation of a matrix $U \in SU(2N+1)$ is just the matrix itself. But they could also form a (reducible) representation of $SU(2N)$, where the representation of $U \in SU(2N)$ is
$$R(U) = \begin{pmatrix} U & \\ & 1 \end{pmatrix}.$$
Similarly they could be a representation of $SU(2N-1)$, or of literally any group, where the representation is $R(U) = I$. This is just the second bullet point above. Note that none of these groups could be called a symmetry group unless you know the representation operators commute with the Hamiltonian.

*Basically, making up representation operators is not useful in physics unless they correspond to a physical operation. In the paper you linked, the claim is that a system of spin $N$ atoms has a symmetry by rotating the hyperfine states into each other; this isn't too different from, e.g. color or isospin rotations. So that's what the $SU(2N+1)$ means, but this is particular to this system they're studying.

A: The representation space of the spin $j$ representation can be thought of as a symmetric collection of $2j$ qubits. For example for $j = \frac{3}{2}$, an orthonormal basis of its $4$-dimensional  representation space is given by:
$$  \begin{array}  \\|0, 0, 0\rangle \\\frac{1}{\sqrt{3}} (|1, 0, 0\rangle 
+|0, 1, 0\rangle +|0, 0, 1\rangle )\\\frac{1}{\sqrt{3}} (|1, 1, 0\rangle 
+|0, 1, 1\rangle +|1, 0, 1\rangle )\\ |1, 1, 1\rangle \end{array}$$
The group $SU(2)$ rotates each qubit equally and rigidly , for $g \in SU(2)$, its action in the above example is given by: 
$$g|v_1, v_2, v_3\rangle  = |gv_1, gv_2,g v_3\rangle$$
where, $v_i$ are the two dimensional vectors defining the qubit state on the Bloch sphere.
The Group $SU(2j+1)$ contains elements which rotate the qubits differently and in addition mix between them.
A: $SU(2)$ is always the symmetry group for particles of any spin, as the generators of spin are rotations. What changes is the dimension of the representation of $SU(2)$ you have to use. For spin-$N$ particles, we use a representation of $SU(2)$ on $\mathbb{C}^{2N+1}$.
A: It is not the same.  A $2N+1$ dimensional representation of SU(2) would have only three generators: $L_+, L_-$ and $L_z$.  In particular, $L_+$ and $L_-$ only connect states with $\Delta m= \pm 1$, i.e. can only connect "neighbouring" states (this is loose terminology).  In addition, the size of the matrix elements are quite restricted as the basic commutation relation $[L_+,L_-]=2L_z$ must be preserved by the representation.
On the other hand, generators of $su(2n+1)$ are of the form $C_{ij}$ with $i,j=1,\ldots, 2n+1$ which have non-zero matrix elements between $i$ and $j$, i.e. these can connect "non-neighbouring" states.  The size of the matrix element of $C_{ij}$ is $1$ and the matrices must satisfy
$[C_{ij},C_{k\ell}]= \delta_{jk}C_{i\ell}-\delta_{j\ell}C_{kj}$.
In other words, yes it is possible to construct a $2n+1$ dimensional representation of $su(2)$ but this representation is NOT the same as the defining $2n+1$ dimensional representation of $su(2n+1)$.  
As a simple example, consider the Gell-Mann matrices for $su(3)$.  You can easily enough construct a $3$-dimensional irrep of $su(2)$ of angular momentum $\ell=1$, but the three matrices for $L_+,L_-$ and $L_z$ do not look anything like the 8 Gell-Mann matrices; it is possible to write some but not all the $su(2)$ generators are linear combos of the Gell-Mann matrices, but it is certainly not possible to write the 8 linearly independent Gell-Mann matrices in terms of the 3 linearly independent angular momentum matrices of dimension $3$.  If anything, there are two diagonal Gell-Mann matrices but only one diagonal $su(2)$ matrix.
