Why use branching rules for $SU(n)$ when classifying states in quantum many-body systems, rather than $GL(n)$? I'm reading Group Theory and its Application to Physical Problems by Hamermesh, and am struggling to understand chapter 11, which is on classifying states of quantum many-body systems, mostly using Young-diagram methods. I also took a look at Symmetry of Many-Electron Systems by Kaplan, which deals with many of the same subjects, but I found in it the same claim which I don't understand.
To state the point I don't understand, let's assume we have a quantum system with hamiltonian $H_0$, which we have already diagonalized. This means we know the decomposition of hilbert space to subspaces $V_E$, parametrized by the energies $E$. The spaces $V_E$ are in general more than 1-dimensional, or in other words, we have a degeneracy associated with each energy level.
We now add a small perturbation $\lambda H_1$ to the hamiltonian: $H_{tot} = H_0 + \lambda H_1$. The subspaces $V_E$ split further, and the degeneracy is reduced. The problem is to figure out how the spaces $V_E$ split to smaller subspaces, each of which is an eigenspace of the new hamiltonian $H_{tot}$.
As far as I can tell, both Hamermesh and Kaplan propose the same solution to this problem, which is the following. We already know the decomposition $V_E$ corresponding to $H_0$. Look at a specific eigenspace of $H_0$, call it $V_{E_0}$, and say it is of dimension $n$. Note the following two facts:

*

*The group $SU(n)$ acts on $V_{E_0}$ in an obvious way, as the group of all special unitary transformations of this space. This gives a representation of $SU(n)$ on $V_{E_0}$ which is necessarily irreducible, since this is a space of dimension $n$. (note that here, Hamermesh uses the group $SU(n)$ while Kaplan uses $U(n)$. Why they use different groups is part of what I'm trying to understand.)


*Assume the full hamiltonian $H_{tot} = H_0 + \lambda H_1$ commutes with some group of transformations $G$, which is known to us, and is a subgroup of $SU(n)$. (In practice I'm mostly interested in the case where $G=SO(3)$, the group of rotations).
The authors then reason (or at least, this is my impression, as they're both not very clear) that since the new hamiltonian $H_{tot}$ commutes with the representation of $G$, which is a subgroup of $SU(n)$, and might not commute anymore with the representation of $SU(n)$, then the representation of $SU(n)$ should decompose into irreducible representations of $G$. This would decompose $V_{E_0}$ further into subspaces on which the irreducible representations of $G$ are realized, and these subspaces will be eigenspaces of the new hamiltonian, since it commutes with $G$.
Therefore, we should just study how the representation of $SU(n)$ on $V_{E_0}$ decomposes into irreducible representations of $G$, or in other words, the branching rules $SU(n) \rightarrow G$. These branching rules would give the new eigenspaces of the full hamiltonian. Both books then go on to use these branching rules to figure out the states of many-electron atoms, in $L-S$ and in $j-j$ coupling.
My only problem with this argument is that it seems to go through no matter what group we choose instead of $SU(N)$, as long as it acts on $V_{E_0}$ and includes $G$ as a subgroup. We could choose $SU(n)$ (as hamermesh does), $U(n)$ (as Kaplan does), or $GL(n)$; or even $SO(n)$ for that matter, as long as $G \in SO(n)$. Why not choose $GL(n)$ actually - as it seems a somewhat more natural group, given that it is the group of all transformations of $V_{E_0}$? Why is $SU(n)$, or $U(n)$, chosen, and is there a difference between them in this context?

To make this clearer I will illustrate this in the case of $j-j$ coupling in atoms, which is the simplest example given in these books since it does not involve spin states.
We have an atom with a hamiltonian $H = H_{rad} + H_{LS}$, where $H_{rad}$ is radial (depends only on $r$), and $H_{LS}$ is the spin-orbit coupling term. For $H_{rad}$, we know that $n$ and $l$ are two quantum numbers, so we use them to label the states $|n,l>$. $H_{LS}$ is treated as a perturbation, and so each energy level $|n,l>$ now splits into two, corresponding to $j=l+\frac{1}{2}$ and $j=l-\frac{1}{2}$. We can now label each state with three numbers $|n,l,j>$, but the levels are still degenerate (in other words there is no actual state $|n,l,j>$ since there is a whole subspace corresponding to these three numbers).
Now assume we have two electrons in this atom, both with the same $n$, $l$ and $j$. This is allowed since the levels are still degenerate (so the electrons are not in the same total quantum state); However, not all states $|n,l,j>$ are allowed, only the antisymmetric combinations of those. For this reason, we apply the antisymmetrization operator to the space spanned by the tensor products of all the $|n,l,j>$ states (with the given $n$, $l$ and $j$), and obtain its antisymmetric subspace, call it $V_{nlj}$. of dimension $d$. The electrons are allowed to occupy states in this space.
The question now asked is what are the $J$ values allowed for the states contained in $V_{nlj}$ (where of course $\bf \vec{J} = \vec{J_1} + \vec{J_2}$, and $\frac{1}{2}J(J+1)$ is the eigenvalue of $\bf J^2$). This is important because we want to add another perturbation to the hamiltonian, which will couple the two $\bf j$'s. The new perturbation will, however, still commute with the rotations of $SO(3)$. The solution is to decompose the obvious representation of $SU(d)$ on $V_{nlj}$ into irreducible representations of $SO(3)$, and apparently the dimensions of the irreducible representations give the correct $J$'s.
 A: We chose $U(n)$ because reducing the full carrier space $V_{E_0}$ is done using a $U(n)$ change of basis, resulting $V_{E_0}$ now carries a decomposable representation of the subgroup.  A representation is reducible if there is a similarity transformation that will bring it to block diagonal form.  Because every representation of a compact or finite group is equivalent to a unitary representation, the similarity transformation is usually itself unitary.
You probably do this all the time without realizing it.  For instance, when you work in spherical coordinates you effectively reduce the 3d harmonic oscillator states of a given total $n$, which span a $U(n)$ irrep of dimension $\frac12 (n+1)(n+2)$, into its $SO(3)$ invariant subspaces using spherical harmonics.  For instance the subspace of states with $n=2$ is 6-fold degenerate, and contains angular momentum states with $L=2$ and $L=0$.
If you start with a basis in Cartesian coordinates $\{\vert n_1n_2n_3\rangle, n_1+n_2+n_3=6\}$, you can't do the required change of basis by limiting yourself to - say - $SO(6)$ transformations.  You could use $GL(6,\mathbb{C})$ but you wouldn't be guaranteed that the resulting change of basis would preserve lengths and you'd have to renormalize the basis vectors in each subspace at the end anyways.
One could use $SU(n)$ instead of $U(n)$.  $U(n)$ contains elements with determinant $\ne 1$ so the transformation is a little more general.  Within each subspace the extra phase that comes with $U(n)$ can be removed "by hand'' if required.
A simple example is provided with the 2-dimensional representation of the rotations about one axis.  An element $R(\theta)$ is given by
$$
R(\theta)\mapsto \Gamma(\theta)=\left(\begin{array}{cc}
\cos\theta&-\sin\theta\\
\sin\theta&\cos\theta
\end{array}\right)\, .
$$
Now this group is Abelian so its irreducible representations are necessarily one-dimensional.  Clearly the complex unitary transformation
$$
U=\left(\begin{array}{cc} 
a&b\\
ia&-ib
\end{array}\right)\, ,\quad U^{-1}=\frac12\left(\begin{array}{cc}
\frac1a&-\frac{i}a\\
\frac1b&\frac{i}b
\end{array}\right)\, ,\qquad a,b\in\mathbb{R}
$$
block-diagonalizes $\Gamma$ for any $a,b$:
$$
U^{-1}\cdot\Gamma\cdot U=
\left(\begin{array}{cc}e^{-i\theta}&0\\
0&e^{i\theta}
\end{array}\right)\, .
$$
Here, $U\in Gl(2,\mathbb{C})$ but the carrier space for each irreducible subspace is spanned by vectors
$$
\vert -\rangle=a\left(\begin{array}{c} 1\\ i\end{array}\right)\, ,\quad \vert +\rangle=b\left(\begin{array}{c} 1\\ -i\end{array}\right)
$$
which only have length 1 if $a=\pm b=\frac1{\sqrt{2}}$.  With this condition the matrix $U$ becomes unitary.
Now you want to be a little careful about using $SO(3)$ is you have spin because the spin degree of freedom commutes with $SO(3)$ operations and classifying states using "only" $SO(3)$ might not fully reduce your initial space.  For instance, in the coupling of a particle of angular momentum $L=1$ with spin $1/2$, the resulting space is of dimension $6$ and decomposes into a sum of states with $J=3/2$ (dimension 4) and $J=1/2$ (dimension 2).  This is a trivial example where $\ell$ isn't enough.
The sneaky part is you can have spaces - like the space of hydrogen atom states with energy $E_2=-13.6/4$, where the $L=0$ and $L=1$ values of angular momentum can lead to multiple copies of some $J$ subspaces.  In this case you might have to use $L$ to distinguish the copies.
Finally, it's important to realize that fully decomposing in a sum of irreducible representations "forces you" into unnatural choices of basis states.  For instance, the permutation group acts naturally by permuting - say - (apple, orange, banana), each fruit considered as a basis vector of a 3-dimensional space,  but this gives a 3-dimensional reducible representation.  The basis states for the irreducible representations in this space are linear combinations of fruits with irrational and not necessarily positive coefficients, whatever that means.
Additional sources:

*

*The original paper of J.P. Elliott


Elliott JP. Collective motion in the nuclear shell model. I. Classification schemes for states of mixed configurations. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences. 1958 May 6;245(1240):128-45.

was the first to exploit the symmetries of the many-body 3d harmonic oscillator problem.  It contains details on reduction from $SU(n)$ to $SU(3)$ to $SO(3)$, and uses Young diagrams.  Hamermesh is written in the same style as this paper.  A more modern treatment can be found in chapters and appendices of

Rowe DJ, Wood JL. Fundamentals of nuclear models: foundational models. World Scientific; 2010.

Dealing with finite groups is sometimes easier and the treatment in

Dresselhaus MS, Dresselhaus G, Jorio A. Group theory: application to the physics of condensed matter. Springer Science & Business Media; 2007

is very nice.  Another classic is

Tinkham M. Group theory and quantum mechanics. Courier Corporation; 2003.

Finally a text with lots of technical details and examples, which I also like, is

Ping J, Wang F, Chen JQ. Group representation theory for physicists. World Scientific Publishing Company; 2002

