In chapter 8.2.3 of Schwartz' textbook "Quantum Field Theory and the Standard Model", the author states the following,
Finally, we expect from representation theory that there should only be two polarizations for a massless spin-1 particle, so the spin-0 and the longitudinal mode should somehow decouple from the physical system.
How does "representation theory" tells us about the polarizations of a massless spin-1 particle? Could someone please explain this statement?
Short answer: Read Weinberg, Vol I, Section 2.5.
Longer answer, hopefully covering more of what Weinberg takes for granted:
In relativistic physics, the Hilbert space is a representation of the Poincare group; rotations, translations, and boosts act on the state vectors linearly, turning one state into another. Representation theory is useful here because it can tell us how to break the Hilbert space up into more understandable pieces.
Suppose we've got a Hilbert space describing the states of some complicated system of many particles, and that we restrict our attention to the subspace of states of a single particle. This subspace must still be a Hilbert space, and it must still carry a unitary representation of the Poincare group, because if we rotate our system, or boost it, or translate it, it's still the same particle.
The pioneers of quantum physics reasoned that if the particle could not be broken apart further, then this Hilbert space must be irreducible. Wigner is frequently credited with this observation, but if you read his paper, he says Dirac made the connection. Majorana seems to have also investigated the idea even earlier, but I'm unable to read his paper.
An irreducible representation is one which doesn't have any proper invariant subrepresentation. This means that if you start with one state vector in an irreducible Poincare representation, you can get to all of the others just by acting with the Poincare group. An irrep contains exactly those states which are needed for Poincare covariance, as determined by anything that's intrinsic to the particle, like spin or momentum states.
In quantum physics, we really only care about projective representations, because states are rays rather than vectors. But a projective representation of a group is just a representation of the group's universal cover. E.g., projective representations of the rotation $SO(3)$ are representations of its double cover $SU(2)$.
So the natural question, which Wigner answered (subject to unitarity and non-negative energy), is "What are the irreducible representations of the Poincare group's universal cover?"
Wigner adapted Frobenius's "little group" method to answer this question. He observed that the Hilbert space breaks up into a direct sum of momentum eigenspaces $$ \mathcal{H} = \oplus_{p \in C} \mathcal{H}_p $$ For irreducible representations, the direct sum is over the 'mass shell' $C$ of all 4-momenta which can be obtained from some given reference momentum $k$ by Lorentz transformations. (This is necessary, and any more would ruin irreducibility.)
Each $\mathcal{H}_p$ is a representation $R$ of the "little group" $W_k$ of Lorentz transformations which fix $k$. Since any momentum $p\in C$ can be written as $p=Lk$ for some Lorentz transform, we can write any the action of any Lorentz $\Lambda$ as transformation as a shift $p\mapsto \Lambda p$ followed by the action of a little group element $W(\Lambda,p)$. Since the $\Lambda$ action is transitive on $C$, a representation of Lorentz is irreducible if and only if the little group representation $\mathcal{H}_k$ is. (Extension to Poincare is easy. Translations have to act by $e^{iap}$ on the eigenspace $\mathcal{H}_p$.)
In the massive case, the obvious 4-momentum to use is $k=(m,0,0,0)$, a particle at rest. The transformations that fix this 4-momentum are just the spatial rotations, and the irreps $\mathcal{H}_{k}$ must be one of the usual finite-dimensional $SU(2)$ reps, labelled by spin $j$.
In the massless case, things are slightly trickier, since massless particles never stop moving. So instead of $k=(m,0,0,0)$, we use $k=(E,E,0,0)$. In this case, the little group is actually a copy of $ISO(2)$, the group of translations and rotations of Euclidean 2-space. The irreps of this are well known (and can be gotten by doing the little group trick again): $ISO(2)$ is a semi-direct product of rotations and translations -- and its irreps are labelled by a discrete 'helicity' (an 'angular momentum-like' label that governs the action of the rotations of the 2-space) and two continuous 'momentum-like' labels (which govern the action of the translations.)
This is where things get weird, and where representation theory doesn't help. We've never seen any massless particles that have continuous labels, so we simply restrict our attention to the case where the act trivially. What remains are one-dimensional representations labelled by the helicity parameter. Lorentz transformations can mix helicity $+j$ with helicity $-j$, so the Poincare irreps we construct are all momentum translates of the two helicity states, which (for $j=1$) transform in just the right way to be polarizations.
So, it's only sort of true that representation theory tells you what the polarizations are. You have to assume the polarizations are not continuous all by yourself.
