Photon is a spin-1 particle. Were it massive, its spin projected along some direction would be either 1, -1, or 0. But photons can only be in an eigenstate of $S_z$ with eigenvalue $\pm 1$ (z as the momentum direction). I know this results from the transverse nature of EM waves, but how to derive this from the internal symmetry of photons? I read that the internal spacetime symmetry of massive particles are $O(3)$, and massless particles $E(2)$. But I can't find any references describing how $E(2)$ precludes the existence of photons with helicity 0.
It derives not from the internal symmetry itself but from the fact that it is a gauge symmetry.
Your symmetry group assignments are not those of the symmetry group but of the little group of the representation. If you assume in addition that the representation is irreducible, you end up in the massless case (with little group ISO(2)=E(2)) with a helicity representation, which picks up from a vector representation only the transversal part, corresponding to a gauge symmetry. Because of reflection symmetry (parity), there are two helicity degrees of freedom. Under the connected part of the Poincare group, this splits into two irreducible representations of fixed helicity, corresponding left and right circular polarization.
This is described in full detail in Section 5.9. of the quantum field theory book (Part I) by Weinberg. In particular, the 2-valuedness (rather than the 3-valuedness) of the helicity is discussed after (5.9.16).