Gauge fixing, invertibility and Green's functional consider the photon in QED and the corresponding EOM of its Green's functional in k-space: $$(k^\mu k^\nu-k^2g^{\mu\nu})\Delta_{\nu\rho}(k)=i\delta^\mu_\rho.$$
Now, I understand that $U^{\mu\nu}(k):=k^\mu k^\nu-k^2g^{\mu\nu}$ is not injective, since $U^{\mu\nu}k_\nu=0$ and thus $\det U=0$. That is why $U$ is not invertible.
In the literature I read that gauge fixing solves this problem. Using the $R_\xi$ gauges, one then obtains a new $U^{\prime\mu\nu}=(1-\xi^{-1})k^\mu k^\nu-k^2g^{\mu\nu}$. It follows that $U^{\prime\mu\nu}k_\nu=-\xi^{-1}k^2k^\mu$ and thus $k_\nu$ does not have the eigenvalue zero anymore.

  
*
  
*How can we be sure that there aren't any other vanishing eigenvalues? Why don't we diagonalise the operator?
  

Also, I remember that in scalar field theory we solved the invertibility problem by analytical continuation and then Feynman-shifting the poles away from the real axis: $p^2-m^2 \mapsto p^2-m^2+i\epsilon.$
We can do the same here, can't we? If we write $U^{\prime\mu\nu}=k^\mu k^\nu-k^2g^{\mu\nu}+i\epsilon$, then we arrive at $U^{\prime\mu\nu}k_\nu=i\epsilon\neq0$ for $\epsilon>0$.


  
*Why do we need gauge fixing to make $U$ invertible? Why isn't it sufficient to analytically continue the operator and then Feynman-shift its poles, as we do in scalar field theory?
  

 A: *

*That is indeed a good exercise that any serious student of QFT should do at least once. In momentum space, the trick is to decompose vectors in components parallel and perpendicular to $k^{\mu}$. 

*Unlike for scalar theory, in the case of gauge theories, the  $i\epsilon$-prescription alone is not enough to render the path integral well-defined without gauge-fixing. (Don't forget that at the end of the day we should take the limit $\epsilon\to 0^+$.) See also e.g. this & this Phys.SE posts.
A: In covariant notation Maxwell's equations can be written as $\partial_\mu ( \partial^\mu A^\nu - \partial^\nu A^\mu) =-j^\nu / \epsilon_0 $. This equation does not fix a one on one relation between field an source so it cannot be inverted. The cause is gauge invariance. To find the Green's function you need to invert the equation of motion. This is not possible for Maxwell's equations because they are gauge invariant, so physicists "fix the gauge" and then argue that the final results are independent of the particular choice they made.
For a valid non gauge invariant theory of electromagnetism see my paper at https://arxiv.org/abs/physics/0106078. 
