Quillen generalized the definition of the determinant of an oparator to
a form applicable to operators with zero modes, between finite or infinite dimensional
Hilbert spaces:
$D: \mathrm{H_1} \rightarrow \mathrm{H_2}$
According to this generalization, the determinant is not
a C-number but an element of a one dimensional vector space :
$\mathrm{Det}(D) = (\wedge^{top}( \mathrm{H_1}/\mathrm{ker}(D)))^{\dagger} \wedge^{top}\mathrm{img}(D))$
Where $\wedge^{top}$ denotes the top wedge product. This basically means that we do not include the zero modes in the
eigenvalue product. For example consider a three dimensional matrix $A$
without zero modes, then its determinant according to Quillen is:
$\mathrm{Det}(A) = e_1^{\dagger} \wedge e_2^{\dagger} \wedge e_3^{\dagger} \wedge A e_1 \wedge A e_2 \wedge A e_3 = \mathrm{det}(A) e_1^{\dagger} \wedge e_2^{\dagger} \wedge e_3^{\dagger} \wedge e_1 \wedge e_2 \wedge e_3 $
Where $\mathrm{det}$ is the conventional matrix determinant. Notice that the
Quillen determinant in this case is just the conventional determinant
multiplied by the one dimensional unit vector $e_1^{\dagger} \wedge e_2^{\dagger} \wedge e_3^{\dagger} \wedge e_1 \wedge e_2 \wedge e_3 $.
Now, it is not difficult verify that the determinant of a diagonal matrix $ A = \mathrm{diag} [ \lambda_1, \lambda_2, , 0]$ with
zero eigenvalues will be just the product of its nonvanishing
eigenvalues times the unit vector composed from the top wedge product the
einvectors with nonvanishing eigenvalues:
$\mathrm{Det}(A) =\lambda_1 \lambda_2,e_1^{\dagger} \wedge e_2^{\dagger} \wedge e_1 \wedge e_2 \equiv det^{'}(A) e_1^{\dagger} \wedge e_2^{\dagger} \wedge e_1 \wedge e_2$
Where $ det^{'}(A)$ is the determinant on the subspace excluding the zero modes. Please notice that now $e_3$ disappeared from the top wedge product.
Relation to anomalies:
The scalar value $\lambda_1 \lambda_2$ of the Quillen determinant is
basis dependent, because if one applies a unitary transformation:
$ A \rightarrow U^{\dagger} A U$
Only the full top wedge product $e_1^{\dagger} \wedge e_2^{\dagger} \wedge e_3^{\dagger} \wedge e_1 \wedge e_2 \wedge e_3 $ is invariant
but not the partial one: $e_1^{\dagger} \wedge e_2^{\dagger} \wedge e_1 \wedge e_2 $ .Thus the scalar value of the determinant changes.
Thus in this case:
$\mathrm{Det}(U^{\dagger} A U) =c(A, U) \mathrm{det^{'}}(A)e_1^{\dagger} \wedge e_2^{\dagger} \wedge e_1 \wedge e_2 $
Where $c(A, U) $ is a scalar depending on $A$ and $U$. Consequently, the
Quillen determinant is not invariant under unitary transformations.
Applying two consecutive unitary transformations one observes that the
additional scalar must satisfy the relation:
$ c(A, UV) = c(V^{\dagger} A V, U) c(A, V)$
This relation is called the one cocycle condition.
This phenomenon occurs when $\mathrm{D}$ is a Dirac operator in the
background of a gauge field. Due to the fact that there exist zero modes,
a unitary transformation on the spinors and the gauge fields gives rise to
a scalar multiple to the determinant stemming from the anomaly.
Basically, there is one type of function of a gauge field and a unitary
operator which satisfies the one cocycle condition (up to a constant
multiple).
Please see the following lecture notes and the following article by M. Blau for further reading.