My understanding is that gauge symmetries are fake in that they are only redundancies of our description of the system that we put in (either knowingly or unknowingly) see Gauge symmetry is not a symmetry?. Moreover it seems like we could in principal work without the gauge redundancy, and things would just be a bit nastier at intermediate stages of a calculation. At the same time, people put a great deal of emphasis that gauge symmetries cannot be anomalous. I understand that if we work with the gauge redundancy then we do indeed need it to be anomaly free so that the unphysical polarizations will cancel. However, what if we work without the gauge redundancy, with just the physical polarizations, what goes wrong then if we don't make sure the would-be gauge symmetry is not anomalous?
If gauge symmetries are not fake, but are real symmetries, then， when they are anomalous, it simply means that the theory just does not have the symmetries. The theory is still well defined, at least.
If gauge symmetries are fake and representing redundancy in our description, then， when they are anomalous, it means that the theory is inconstant.
So because gauge symmetries are fake, that makes us do care if they are anomalous?
This is a very natural question I wondered some time ago.
In electrodynamics, if you want to work just with two physical polarizations, you have two possibilities:
Is it then impossible to work without gauge invariance? I would just say that we do not know how to do it, and it seems difficult.
In short, to answer the OP quoted question, what goes wrong is either the loss of Poincare invariance or the $1/r$ electrostatic potential.