As is known, the presence of gauge anomalies leads to breakdown of the unitarity of the gauge theory.
One way to understand this is to involve the BRST quantization of the gauge field theory. It reads by the following way. The gauge invariance describes in fact the redundance of the Hilbert space of gauge variant rays to the space of gauge-invariant rays. In the result, the correct state in gauge theory is defined to be invariant under the gauge transformation. In path integral formulation of the gauge theory, this redundancy is nothing but reduction the integration over all gauge fields configuration to the integration over ones satisfying the gauge fixing condition; the latter defines a surface (gauge orbit) in space of gauge fields configurations.
For particular choices of the gauge fixing condition this redundancy leads to generating the ghosts action. The ghoats are unphysical states with indefinite norm in the Hilbert space. Although they mediate the physical processes, they can't be in in- or out- states, so their indefinite norm doesn't make the unitarity to be broken. Their ability to contribute to the physical state is forbidden by Slavnov-Taylor identities; the latter are direct consequence of underlying gauge invariance.
If, however, the gauge anomaly is present, then the Slavnov-Taylor identities are broken. Therefore the ghosts contribute in the Hilbert space of physical states, and the unitarity is broken.
It is always possible to choose the gauge fixing in a way that ghosts don't present. In abelian gauge theories an example is Lorentz gauge. In non-abelian gauge theories, an example is the so-called auxillary gauge. With this choice of gauge fixing conditions, there are no intermediate states with indefinite norm whose presence leads to the violation of the unitarity in a case of the gauge anomaly. So where exactly the unitarity breakdown is hidden in the case of fixing the gauge condition in a way such that the ghosts are absent?
In fact, although the gauge invariance says us that all gauge fixing conditions are equivalent, and one might say that the unitarity has to be preserved for all possible choices. However, I may say that the gauge anomaly requires the quantization by using the ghost-free choices of fixing condition, so that the unitarity is preserved (as long as I don't see where the unitarity breakdown is hidden).