I'm not entirely sure that I've understood what you're referring to, but it sounds like the following. To figure out the allowed representations for a massive particle in $D$-dimensional space-time dimensions, we can boost to its rest frame. Then we have the remaining $SO(D-1)$ rotations that leave us in the rest frame of the particle, and so its the respresentation of $SO(D-1)$ that determine the properties of the particle. So in $D = 4$ our massive particles are labelled by representations of $SO(3)$, which is good old angular momentum.
On the other hand if we have a massless particle, we can't boost to its rest frame since it doesn't have a rest frame. So what we have to look at instead are the transformations that leaves it's direction fixed, that is the transformations that fix a null ray. This is $SO(D-2)$ (if we ignore the fact that one of the directions is not compact, I think its really $SO(D-3,1)$ or whatever it's called), and so it is the representations of $SO(D-2)$ that determine the states of massless particle. In $D = 4$ we have that our massless particles are labeled by representations of $SO(2)$. So when we say the photon has angular momentum 1 we are not talking representation of $SO(3)$. It does not have a state with $m = 0$ - a longitudinal polarization - even though the $l=1$ represenation of $SO(3)$ has three states $m=-1,0,1$.
I hope this is what you were looking for.