Arguments from the "little group" are used to show that the internal degrees of freedom of a massive particle transform under $SO(3)$, while the internal degrees of freedom of massless particles transform under $E(2)$. This sort of argument shows us that a massless particle can only have two degrees of freedom. Meanwhile, a spin-1 field (e.g. a photon) will typically be represented via a four vector $A_{\mu}$ at every point in spacetime, and a spin-2 field (i.e. the graviton) will be represented as a two-tensor $g_{\mu\nu}$, so that the fields can transform consistently under Lorentz transformations. There is an obvious conflict between the number of "physical" degrees of freedom of such particles as described by the little group argument, and the number of degrees of freedom in such a description of the field. To rectify this we assert that all but two of such degrees of freedom are either "pure gauge" or nondynamical, so that only the physical degrees of freedom play any part in the dynamics of our theory.
My question is: without knowing the Lagrangian a priori, is there a way of deriving the gauge group for a spin-1 or spin-2 field ($A_\mu \to A_\mu + \partial_\mu f$ for the spin 1 case and the diffeomorphism group for $g_{\mu\nu}$) strictly by knowing the "true" degrees of freedom (i.e. the transverse polarization modes/helicity states)?
