Is there such thing as a 3D Chern invariant (or some other quantity) that I can use to test an insulating quasiparticle spectrum is a topologically trivial or non-trivial insulator?
Does one exist for a state with broken time reversal symmetry, i.e. a chiral spin liquid?
Specifically I'm looking at a 3D condensed matter system, where a chiral spin liquid is assumed and a gap opens when I add certain terms.
Yes, there is an bulk invariant for 3D topological insulators known as the second Chern parity $P_3$ [1-3], as the integral of the Chern-Simons 3-form of the (presumably non-Abelian) Berry connection $\mathcal{A}$ (in the momentum space) over the Brillouin zone (BZ). Note that now the Brillouin zone is a 3 dimensional manifold (as a 3D torus).
$$P_3=\frac{1}{16\pi^2}\int_\text{BZ}\mathrm{Tr}(\mathcal{F}\wedge\mathcal{A}-\tfrac{1}{3}\mathcal{A}\wedge\mathcal{A}\wedge\mathcal{A}),$$
where $\mathcal{F}=\mathrm{d}\mathcal{A}+\mathcal{A}\wedge\mathcal{A}$ is the Berry curvature. The Berry connection $\mathcal{A}$ can be obtained from the Bloch wave functions in the occupied bands. Let $|n k\rangle$ be the Bloch wave function of electron in the $n$th band at the quasi-momentum $k$ (here $k=(k_x,k_y,k_z)$ is a 3-component vector). $\mathcal{A}$ is restricted to the subspace of occupied bands, i.e. we only take those $n$ 's such that the single-particle energies $\epsilon_{nk}<0$ are negative.
$$\mathcal{A}_{mn}(k)=-\mathrm{i}\langle mk|\mathrm{d}|nk\rangle,$$
where the differential operator $\mathrm{d}=\partial_{k_\mu}\mathrm{d}k_\mu$ is defined in the momentum space. It was proved that $P_3$ can only be an integer or a half-integer. If $P_3=0$(mod 1), then the insulator is trivial. If $P_3=\tfrac{1}{2}$(mod 1), then the insulator is topological. In fact, $(-1)^{2P_3}$ is the $\mathbb{Z}_2$ index of the 3D topological insulator. There are other equivalent expressions for $P_3$ which can be found in [1-3] and the references therein.
In 3D, all the fermionic symmetry protected topological (SPT) states needs time-reversal symmetry protection (either $\mathcal{T}^2=+1$ or $\mathcal{T}^2=-1$). So there is not a topological nontrivial state which can also be chiral. I think the chiral spin liquid you are looking for does not exist, but $Z_2$ or $U(1)$ spin liquids with topological spinon band structures and anomalous gapless spinon surface modes do exist, and have been discussed a lot recently.
[1] https://physics.aps.org/featured-article-pdf/10.1103/PhysRevB.78.195424
[2] http://arxiv.org/pdf/1004.4229.pdf
[3] http://journals.aps.org/prx/pdf/10.1103/PhysRevX.2.031008
