Gauge anomalies ruin the unitarity - the explanation involving ghosts An outline
As is known, the presence of gauge anomalies leads to breakdown of the unitarity of the gauge theory. 
One way to understand this is to involve the BRST quantization of the gauge field theory. It reads by the following way. The gauge invariance describes in fact the redundance of the Hilbert space of gauge variant rays to the space of gauge-invariant rays. In the result, the correct state in gauge theory is defined to be invariant under the gauge transformation. In path integral formulation of the gauge theory, this redundancy is nothing but reduction the integration over all gauge fields configuration to the integration over ones satisfying the gauge fixing condition; the latter defines a surface (gauge orbit) in space of gauge fields configurations. 
For particular choices of the gauge fixing condition this redundancy leads to generating the ghosts action. The ghoats are unphysical states with indefinite norm in the Hilbert space. Although they mediate the physical processes, they can't be in in- or out- states, so their indefinite norm doesn't make the unitarity to be broken. Their ability to contribute to the physical state is forbidden by Slavnov-Taylor identities; the latter are direct consequence of underlying gauge invariance.
If, however, the gauge anomaly is present, then the Slavnov-Taylor identities are broken. Therefore the ghosts contribute in the Hilbert space of physical states, and the unitarity is broken.
My question
It is always possible to choose the gauge fixing in a way that ghosts don't present. In abelian gauge theories an example is Lorentz gauge. In non-abelian gauge theories, an example is the so-called auxillary gauge. With this choice of gauge fixing conditions, there are no intermediate states with indefinite norm whose presence leads to the violation of the unitarity in a case of the gauge anomaly. So where exactly the unitarity breakdown is hidden in the case of fixing the gauge condition in a way such that the ghosts are absent?
In fact, although the gauge invariance says us that all gauge fixing conditions are equivalent, and one might say that the unitarity has to be preserved for all possible choices. However, I may say that the gauge anomaly requires the quantization by using the ghost-free choices of fixing condition, so that the unitarity is preserved (as long as I don't see where the unitarity breakdown is hidden).
 A: In the classical theory, the gauge symmetry is necessary for removing the unphysical deegres of freedom of a given gauge field (sometimes also called ghosts).
A massless vector field should only have 2 phyiscal deegres of freedom. Yet, if you think of QED, without the gauge fixing condition the photon still has 3 d.o.f., one of these with zero norm, therefore spoiling unitarity. The decoupling of these unphysical d.o.f. in the quantized theory is ensured by the Ward identites (or the Slavnov-Taylor identites in the non-Abelian case). In QED imposing a gauge condition does not break the conservation of the electromagnetic current, which is a source of the gauge field, and the Ward identities follow directly from the conservation of this current. An anomalous symmetry spoils the conservation of the associated current and the Ward identities break down. Therefore, in an anomalous QED the photon would couple with 3 deegres of freedom to physical processes, which is of course inconsistent with reality.
The Faddeev–Popov ghosts you are referring to arise in the non-Abelian case because there the gauge fixing condition breaks the conservation of the associated current automatically and therefore, no Ward identity can be constructed there. In order to remove the unphysical deegre of freedom from our gauge field, we need to modify our Lagrangian appropriatly, which introduces the Faddeev-Popov determinant and therefore the ghosts. These ghost states ensure the decoupling of the unphysical d.o.f. but of course they are themselves not physical as they are only introduced by a choice of gauge fixing. Therefore the Slavnov-Taylor identites guarantee their decoupling as external states. These can also be derived from the conservation of a specific supersymmetric current in the BRST system by the way and an anomaly spoils the conservation of this BRST current.
Of course you could also choose a gauge in which the ghosts decouple from the gauge field completely and therefore can just be put in the normalization of the PI. But the price to pay for that is the rather complicated form of the gauge boson propagator which itself has now to guarantee the decoupling of the unphysical d.o.f. . For an explicit calculation see my links below.
Moreover, there can be some topological obstacles for choosing this auxilliary gauge globally over a non-trivial base manifold, since this is equivalent to a globally defined section in the gauge bundle (which does not need to exist for non-trivial bundles).
Summary: Unitarity breaks down in presence of an Anomaly because the unphysical (zero-norm) deegres of freedom of the gauge field don't decouple anymore. The introduction of Faddeev-Popov ghosts only "shifts" the unphysical d.o.f. to these new states, therefore they have to decouple in an anomaly free theory.
Sources:
The Problem of the existence of the globally defined section, known as Gribov ambiguity
A book in which the calculation of the propagator for the axial gauge and the decoupling of unphysical d.o.f. of the gauge boson is shown
A paper which i wasn't able to find in the internet by Kummer, W.
(1976) Acta Phys. Austriaca Suppl. XV, 423.
