An open algebra for a collection of first-class constraints, $G_a$, $a=1,\cdots, r$, is given by the Poisson bracket $\{ G_a, G_b \} = {f_{ab}}^c[\phi] G_c$ classically, where the structure constants are functions of the dynamical degrees of freedom, $\phi$. When quantizing a gauge theory, a physical state $|\psi\rangle$ has to satisfy the first-class constraints $\widehat{G}_a |\psi\rangle = 0$. From this, one can easily see $[\widehat{G}_a, \widehat{G}_b]|\psi\rangle = 0$. In the quantum version of the theory, the Poisson equation has to be replaced by an operator commutator equation. In general, ${\widehat{f}_{ab}^c}[\widehat{\phi}]$ doesn't commute with $\widehat{G}_c$. One possibility is the right hand side of the equation for the commutator of two constraints is ordered so that the constraint $\widehat{G}_c$ is always on the right in the operator product. However, the resulting product will be nonhermitian in general due to noncommutativity. The commutator of two Hermitian operators is always antihermitian. So, this means the first-class constraint operators have to be nonhermitian. If we want the constraint operators to be hermitian, we require $[\widehat{G}_a, \widehat{G}_b] = i O(\widehat{f}_{ab}{}^c[\widehat{\phi}]\widehat{G}_c)$ where $O$ is some form of operator ordering. However, this operator ordering will in general contain some terms which don't annihilate $|\psi\rangle$ in general because $\widehat{G}_c$ won't always be on the right. How does one get around this?
3 Answers
I) Let us reformulate OP's question(v1) as
How can hermiticity$^1$ be maintained for the gauge algebra $$\tag{1} [\hat{G}_a , \hat{G}_b ] ~=~ i\hbar~\hat{G}_c ~\hat{f}^{c}{}_{ab} $$ of first-class operator constraints $\hat{G}_a$, if the structure operators$^2$
$$\tag{2} \hat{f}^{c}{}_{ab}~=~f^{c}{}_{ab}(\hat{q}^i,\hat{p}_j)$$ depend on the phase space operators $\hat{q}^i$ and $\hat{p}_j$?
(Note that on the r.h.s. of eq.(1), we let the operator $\hat{G}_c$ stand to the left of the operator $\hat{f}^{c}{}_{ab}$. This is done for purely conventional reasons to follow Ref. 1. This rearrangement just means that we should work with physical bras $\langle \psi |$ rather than physical kets $|\psi \rangle$, which is an equivalent formulation.)
II) Our first point is that the gauge algebra operator identity (1) is just the first in a (possible infinite) tower of operator consistency relations. E.g. the structure operators (2) should satisfy a Jacobi-like operator identity, which in turn involves a new set of higher structure operators, and so forth.
It turns out that the most systematic approach is to recast the gauge symmetry (1) in the Batalin–Fradkin–Vilkovisky (BFV) formalism, which is a generalization of the Hamiltonian BRST method from Yang-Mills theory to arbitrary first-class$^3$ systems (1), even so-called reducible gauge algebras.
The main object in BFV theory is a fermionic BRST charge operator$^4$
$$\tag{3} \hat{Q} ~=~ \hat{G}_a ~\hat{\cal C}^a +\frac{1}{2}\hat{\bar{\cal P}}^c~\hat{f}^{c}{}_{ab}~\hat{\cal C}^b\hat{\cal C}^a +\ldots $$
that squares to zero
$$\tag{4} \hat{Q}^2~=~0. $$
We would for brevity obviously have to leave out a lot of details here, but let us mention that $\hat{\cal C}$ and $\hat{\bar{\cal P}}$ are ghosts and ghost-momenta, which carry ghost number $+1$ and $-1$, respectively. The BRST charge operator $\hat{Q}$ is required to have ghost number $+1$. The gauge algebra (1) is encoded as one of the first operator relations in a (possible infinite) tower of operator relations that are hidden inside the nilpotency condition (4).
The upshot is that the unitarity of the theory is essentially implemented by (among other conditions) requiring Hermiticity of the BRST charge
$$\tag{5} \hat{Q}^{\dagger}~=~ \hat{Q}. $$
Eq. (5) dictates to a large extend what kind of Hermiticity/reality structure that one should impose on the system. In general, these Hermiticity/reality structure conditions will interrelate between the first-class operator constraints $\hat{G}_a$, the structure operators (2), the higher structure operators, etc, cf. Ref. 1.
References:
- I.A. Batalin and E.S. Fradkin, Operatorial quantization of dynamical systems subject to constraints. A further study of the construction, Annales de l'institut Henri Poincaré (A) Physique théorique, 49 (1988) 145. The pdf and djvu files are available here.
$^1$ We will ignore subtleties with unbounded operators, domains, selfadjoint extensions, etc., in this answer.
$^2$ A semantical side remark: The notion of an open gauge algebra is traditionally a notion in Lagrangian formalism, where the gauge algebra is then broken off-shell. In general, it is less straightforward to identify in the Hamiltonian language, if a gauge systems (1) corresponds to an open gauge algebra in the Lagrangian formalism, or if it doesn't.
$^3$ The BFV formalism has since been further developed to deal with second-class constraints.
$^4$ Expansions of $\hat{Q}$ with other operator orderings (Weyl ordering, Wick ordering, etc.) in the ghost sector are possible, see e.g. section 6 in Ref. 1 for further details. The BRST charge operator $\hat{Q}$ is in principle allowed to depend on $\hbar$.
The correct answer makes use of BRST. In short, $\widehat{G}_a$ is nonhermitian in general. Let me explain. In BRST, we augment the gauge and matter fields with ghost fields $\widehat{c}^a$, $\widehat{b}_b$ which satisfy the canonical anticommutation relations $\{\widehat{c}^a, \widehat{c}^b\} = \{\widehat{b}_c, \widehat{b}_d\} = 0$ and $\{ \widehat{c}^a, \widehat{b}_b\}=\delta^a_b$. In addition, we require both ghost fields to be Hermitian. This means the ghost sector has to have an indefinite norm. Define the total ghost number operator as $\widehat{N}_{gh}\equiv \widehat{c}^a\widehat{b}_a$. There is a fermionic operator $\widehat{\Omega}$ with ghost number $+1$, is Hermitian, and is quadratically nilpotent $\widehat{\Omega}^2=0$.
Expand $\widehat\Omega$ as $$\widehat{\Omega} = \widehat{c}^a \widehat{G}_a + \frac{1}{2!}\widehat{c}^a\widehat{c}^b \widehat{b}_c \widehat{f}_{ab}{}^c + \frac{1}{3!2!}\widehat{c}^a\widehat{c}^b\widehat{c}^c\widehat{b}_d\widehat{b}_e\widehat{f}_{abc}{}^{de} + \dots$$ where the $\widehat{G}, \widehat{f}$ operators contain no ghost factors. It's important to observe that $\left(\widehat{c}^a\widehat{c}^b\widehat{b}_c\right)^\dagger = -\widehat{c}^a\widehat{c}^b\widehat{b}_c -\delta^a_c\widehat{c}^b +\delta^b_c\widehat{c}_a$. So, the condition $\widehat{\Omega}^\dagger = \widehat{\Omega}$ translates into infinitely many relations starting with $$\widehat{G}_a=\widehat {G}_a^\dagger-\frac{1}{2}\widehat{f}_{ba}{}^{b}{}^\dagger+\frac{1}{2}\widehat{f}_{ab}{}^{b}{}^\dagger+\dots\,.$$ Anyway, you see the constraints $\widehat{G}_a$ are no longer Hermitian in general.
A physical state satisfies $\widehat{\Omega}|\psi\rangle=0$. If this state has zero ghost number, this reduces to the first class constraint $\widehat{G}_a|\psi\rangle =0$.
It's interesting to observe the special case of quantum gravity in the ADM formalism. There, we have Hamiltonian constraints and diffeomorphism constraints, and they form an open algebra. If we define the extended Hamiltonian as $\widehat{H}^* = \int d^3x\,\{\widehat{b}(x),\widehat{\Omega}\}$ where $\widehat{b}(x)$ is the ghost operator associated with time diffeomorphisms at the spatial point $x$, then the extended Hamiltonian operator is nonhermitian! Replacing it with $\int d^3x\,\{\widehat{ N}(x)\widehat{b}(x),\widehat{\Omega}\}$ where $\widehat{N}(x)$ is some gauge-fixing lapse field operator doesn't change this fact at all.
The answers given basically boils down to drop the Hermiticity condition for $\hat{G}_a$. OK. Let's say classically, the Poisson bracket goes as $\{G_a,G_b\}=f_{ab}{}^c G_c$, and after quantization, we require that this translates into $\left[ \hat{G}_a, \hat{G}_b\right]=i\hat{f}_{ab}{}^c \hat{G}_c$ with the operator product on the right hand side taken in precisely this order. The difficulty is, only very particular choices for the operator product ordering for $\hat{G}_a$ can lead to this form of operator ordering on the right hand side. More accurately, maybe we shouldn't think of it as an ordering product prescription as much as a particular choice of $\hbar$ deformation in quantization. In general, for open algebras, it's going to be very hard to find a $\hbar$ deformation with this property. How does one go about finding a deformation with this property?
