Photon is a spin-1 particle. Were it massive, its spin projected along some direction would be either 1, -1, or 0. But photons can only be in an eigenstate of $S_z$ with eigenvalue $\pm 1$ (z as the momentum direction). I know this results from the transverse nature of EM waves, but how to derive this from the internal symmetry of photons? I read that the internal spacetime symmetry of massive particles are O(3), and massless particles E(2). But I can't find any references describing how E(2) precludes the existence of photons with helicity 0.
It derives not from the internal symmetry itself but from the fact that it is a gauge symmetry.
Your symmetry group assignements are not those of the symmetry group but of the little group of the representation. If you assume in addition that the representation is irreducible, you end up in the massless case (with little group E(2)) with a helicity representation, which picks up from a vector representation only the transversal part, corresponding to a gauge symmetry.
This is described in full detail in the quantum field theory book (part I) by Weinberg.