Quantizing first class constraints

Question

Let $\gamma$ denote a first class constraint. Then if there exists a function on phase space $f(q,p)$ for which the Poisson bracket with the constraint does not vanish $\lbrace f, \gamma\rbrace \neq 0$, then there is an inconsistency with canonical quantization: if we upgrade the constraint equation to an operator relation $\hat{\gamma}=0$ then $\hat{\gamma}$ should annihilate all states whereas canonical quantization states that: \begin{equation} [\hat{f},\hat{\gamma}]=i\hbar {\lbrace f, \gamma\rbrace }. \tag{1} \end{equation} The LHS of Eq.($1$) should annihilate all states as $\hat{\gamma}=0$, whereas the resultant operator on the RHS need not annihilate all states. How does one resolve the inconsistency in this case?

The Dirac bracket does not resolve this as the Dirac bracket of any function with a first class constraint is equal to the Poisson bracket, i.e. $\lbrace f, \gamma\rbrace_{Dirac}=\lbrace f, \gamma\rbrace_{Poisson}$.

Answer 1

The physics lore is that first class constraints$^1$ $\gamma$ generate gauge symmetry. A physical observable $f$ should be gauge invariant. Classically, this means that the observable $f$ should Poisson commute with the first class constraints $\gamma$. Therefore OP's example $f$ with $\{f,\gamma\}\neq 0$ does not correspond to a physical observable.

--

$^1$ Let us for simplicity imagine that there are no second class constraints so that the Dirac bracket is just the Poisson bracket.