How to count the number of modes/polarizations of a Gaussian field theory? A Gaussian (free) field theory is described by a quadratic action of the field, e.g. $S=\int\psi^\dagger K\psi$ (or $S=\frac{1}{2}\int\phi^\intercal K\phi$ for real fields). Usually one just need to diagonalize the action kernel $K$, then each eigen vector corresponds to a mode/polarization of the field. For example,
$$S=\int \psi^\dagger(-i\partial_\tau+H)\psi\stackrel{\text{diag}}=\sum_n\psi_n^\dagger(-\omega+E_n)\psi_n,$$
where each $n$ labels a mode of the field $\psi$, and the dispersion relation (or the energy spectrum) is given by setting the eigen value to zero, such as $(-\omega+E_n)=0$ and hence $\omega=E_n$.
However when I tried to apply this approach to a gauge theory, I got some trouble. For instance, consider the Maxwell theory (with a Euclidian metric in 4 dimension),
$$S=\int\frac{1}{4}F^2=\int\frac{1}{2}A^\mu\Pi_{\mu\nu}A^\nu,$$
where $\Pi_{\mu\nu}=k^2\delta_{\mu\nu}-k_\mu k_\nu$ is the action kernel, and $k_\mu=i\partial_\mu$ is the energy-momentum vector. $\Pi$ is a $4\times 4$ matrix which can be diagonalized. There will be one zero mode, corresponding to the gauge transformation of the gauge field (as can be seen from its eigen vector $A_\mu\sim\partial_\mu \phi$), which should not be counted as a physical mode. So far so good. But there are still three (degenerated) non-zero modes, with the same eigen value $k^2$. At this point, I would tend to conclude that there should be three physical modes, all degenerated on the dispersion relation $k^2=0$. But in fact, photon only have two transverse modes. My question is what is wrong with the mode counting? Shouldn't the longitudinal mode already excluded as the zero mode (gauge mode), but why we are still left with three eigen modes in $\Pi$?

If we perform a pseudo-inverse of $\Pi$, the photon propagator should be
$$-(\Pi^{-1})_{\mu\nu}=\frac{1}{k^2}\left(\delta_{\mu\nu}-\frac{k_\mu k_\nu}{k^2}\right),$$
which also has three poles along the dispersion $k^2=0$. If each pole corresponds to a physical mode, then there will be three photon modes, which is still in contradiction with the fact that photon only have two transverse modes. How to do the mode counting correctly?
 A: The longitudinal mode decouples from all physical processes as a consequence of gauge invariance, which in turn forces the Ward identity
$$ k^\mu \mathcal{M}_\mu = 0$$
where the S-matrix element decomposition $\mathcal{M}^\mu$ is obtained from the polarization vector $\epsilon^\mu(k)$ by $\mathcal{M} = \epsilon^\mu(k) \mathcal{M}_\mu$.
This decoupling (and, additionally, zero-norm) mode is also called a spurious mode. Since it decouples from all physical processes, it does not belong to the physical Hilbert space, and we are left with only two physical polarizations for a massless vector field.
As an aside, a massive vector field does not have gauge invariance, hence no Ward identity, and there the longitudinal mode is not zero-norm, and so massive vector fields indeed have three polarizations.
A: If we write $A_\mu(x)=\varepsilon_\mu(p)e^{ipx}$, the polarization vector should satisfy $\varepsilon_\mu p^\mu=0$, which is a Lorentiz-invariant relation, and is necessary to make sure that we have an irreducible representation of the Lorentz group (actually, the little group that leaves the momentum invariant). This knocks down the number of D.O.F to 3. But still we need to impose gauge invariance $\varepsilon_\mu\sim\varepsilon_\mu+p_\mu$ (otherwise the polarization vector does not transform nicely under Lorentz transformation), so finally we have only two physical polarizations.
These can be made more explicit and rigorous if we study the symmetry transformations of single-particle states more carefully. Weinberg's QFT volume I treats this in great detail, or you can look at http://phys.columbia.edu/~nicolis/GR_from_LI.pdf which follows Weinberg's treatment.
