Orthogonality between the BRST-Closed Subspace and the non-BRST-Closed One Consider the BRST quantisation of free Maxwell theory, in one of the averaged Lorenz gauges,
$$S = \int - \frac{1}{4} F^2 + \frac{1}{2} (\partial\cdot A)^2 + i \bar{C} \partial^2 C.$$
Calling the modes of the ghost $C$ $c$ and the modes of the potential $A_\mu$ $a_\mu$, the state
$$p^\mu a_\mu^\dagger |0\rangle = Q_B\ c^\dagger |0\rangle$$
is BRST-exact, and the state $a_0^\dagger |0\rangle$ isn't, as can be seen by the equation
$$Q_B a_0^\dagger |0\rangle = p_0 c^\dagger |0\rangle.$$
What I'm confused about is that these two states aren't orthogonal,
$$\langle 0| a_0 (p \cdot a^\dagger) |0\rangle = -p^0.$$
Why is this okay?
The reason I'm finding this odd is that I expect a physical Hilbert space to be orthogonal to the extra stuff, since that's the usual understanding of clean separation in vector spaces.
Why is the BRST-closed Hilbert space not cleanly separated in this sense, and in what sense is it cleanly separated (apart from being invariant under time-evolution, of course)?
 A: So I figured it out.
There are two points the question gets wrong:


*

*There is no unique choice of a non-BRST-closed subspace.

*The correctly chosen non-BRST-closed subspace doesn't need to be orthogonal to the entire BRST-closed subspace, only to the particular representative of the BRST cohomology subspace.


For the first point, note that if $|\psi\rangle$ is not BRST closed, neither is $|\psi\rangle + |\chi\rangle$ where $Q_B |\chi\rangle = 0$. In other words, you can add an arbitrary BRST-closed state to a non-closed one and the resultant is still not closed. Because of this, there's no canonical choice of a non-closed subspace.
The nicest way to convince you of the second point is to write down the Hamiltonian for the Lagrangian in the question in terms of oscillators.
Calling the two transverse oscilaltors at any given momentum $\vec{k}$ $a_\alpha$, the oscillator corresponding to $A_0$ $a_0$ and the oscillator corresponding to the spatial longitudinal potential $\vec{\nabla} \cdot \vec{A}$ $a_L$, the Hamiltonian is
$$H = \int d^3 k \left[ \left\{ \sum_\alpha a_\alpha^\dagger a_\alpha \right\} + \left\{ \bar{c}^\dagger c + c^\dagger \bar{c} \right\} + \frac{1}{2} \left\{ (a_0 + a_L)^\dagger (a_0 - a_L) +(a_0 - a_L)^\dagger (a_0 + a_L) \right\} \right].$$
Now, $(a_0+a_L)$ corresponds to the space-time longitudinal mode which is BRST-exact in this gauge (it is the BRST variation of $\bar{c}$) and $(a_0-a_L)$ is the "fourth polarisation," which is not BRST-closed.
The point to take away from that Hamiltonian is, time evolution mixes the non-BRST-closed states with the BRST-exact states!
But, the non-closed states are orthogonal to the cohomology representatives, which are the transverse oscillators, and time-evolution doesn't mix them, so the physics is fine.

The reason I was worried about this question was that I was wondering about gauge-invariance.
Changing gauge corresponds to choosing a different set of states as the representatives of the BRST cohomology. The fact that the non-closed states need to decouple in every gauge then seemed to mean that they better be orthogonal to any representative you can choose, and by changing gauge you can make basically any state in the BRST-closed subspace the representative (I thought, this is also wrong but not in an interesting way), so the non-closed states better be orthogonal to the entire closed subspace.
The point is this: changing gauge also means choosing a different representative for the non-closed subspace that is orthogonal to the new representative of the cohomology subspace!
The reason no one feels the need to point this out is that the exact form of the unphysical states isn't terribly relevant, seeing as they're unphysical and they're guaranteed to decouple by more general considerations anyway.
As a parting comment, it seems to me (but I haven't checked properly) that the way to figure out a* correct representative for the non-closed states is to find the analog of the Hamiltonian above and just observe what the oscillator multiplying the exact oscillator is. Doing the correct quantisation can be subtle, but the point is that the exact non-closed mode that decouples into the Kugo-Ojima quartet (the analog of the set $\{c,\bar{c},a_0+a_L,a_0-a_L\}$ in whatever gauge) is the one that needs to be orthogonal to the physical states.
*Adding a BRST-exact state to any non-closed state doesn't change its inner product with any closed state, so there is no canonical choice.

To read more about the inner product from the place where I found the answer, see Henneaux and Teitelboim's Quantisation of Gauge Theories, section 14.2.
