Wigner vs. BRST approach to Klein-Gordon In Wigner's classification of particles Wigner, E. (1939). On Unitary Representations of the Inhomogeneous Lorentz Group. Annals of Mathematics, 40(1), 149–204. http://www.jstor.org/stable/1968551 the Hilbert space of a single spinless particles of mass $m$ is given by functions $\tilde{\varphi}$ of 4-momentum $p$ which are supported on the mass shell $p^2=m^2$ and have inner product
$$\langle\tilde{\varphi}_1,\tilde{\varphi}_2\rangle=\int \frac{d^3\mathbf{p}}{2E_\mathbf{p}}\tilde{\varphi}_1^*(p)\tilde{\varphi}_2(p)|_{p^0=E_\mathbf{p}}.$$
In particular, in terms of the position space
$$\varphi(x)=\frac{1}{(2\pi)^{3/2}}\int\frac{d^3\mathbf{p}}{2E_\mathbf{p}}e^{-i p\cdot x}\tilde{\varphi}(p)|_{p^0=E_\mathbf{p}},$$
the inner product is
$$\langle \varphi_1,\varphi_2\rangle=i\int d^3\mathbf{x}\left(\varphi_1^*\frac{\partial\varphi_2}{\partial t}-\frac{\partial\varphi_1^*}{\partial t}\varphi_2\right)$$
at any time slice. These position space functions satisfy the Klein-Gordon equation since the momentum space ones where concentrated on the mass shell. It is usually claimed that this cannot be interpreted as a probability amplitude for position given that this inner product is not positive definite. Moreover, in this scheme there doesn't seem to be any plausible notion of position operator (see Ticciati, R. (1999). Quantum Field Theory for Mathematicians. Cambridge University Press section 1.6). A nice summary of these facts is in the first chapter of Haag, R. (1996). Local Quantum Physics (2nd ed.). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-61458-3.
On the other hand, in section 4.2 of Polchinski, J. (1998). String Theory. In Cambridge Monographs on Mathematical Physics. Cambridge University Press. https://doi.org/10.1017/CBO9780511816079, there is another approach to the quantum theory of a single spinless particle. Instead of being based on a classification of the representations of the Lorentz group, it is given by quantization of the classical theory of the particle
$$S=\int d\tau\left(\frac{1}{2e}\dot{x}^\mu\dot{x}_\mu-\frac{1}{2}em^2\right).$$
The BRST procedure here yields the bosonic operators $x^\mu$ and $p^\mu$ satisfying $[x^\mu,p^\nu]=i\eta^{\mu\nu}$ fermionic operators $b$ and $c$ satisfying ${b,c}=1$, and a BRST operator $Q=cH$, where the Hamiltonian $H=\frac{1}{2}(p^2+m^2)$. Then the Stone - von Neumann theorem says that the Hilbert space of the theory is given by $\mathbb{C}^2$ valued functions $\varphi$ of position $x$ with inner product
$$\langle\varphi_1,\varphi_2\rangle=\int d^4x\varphi_1^*(x)\varphi_2(x).$$
After taking the BRST cohomology, one can essentially forget that they are $\mathbb{C}^2$ valued and one obtains the additional restriction that they must satisfy the Klein-Gordon equation. However, unlike the previous case, in this perspective there is a position operator and one can interpret $\varphi$ as the spatial (spacetime) probability amplitude of the particle.
I am very confused by the fact that these two approaches give different answers. What is the relationship between them?

A comment:
The Klein-Gordon QFT yields a Hilbert space which is the second quantization of the one described by Wigner. On the other hand, the Feynman diagram expansion of the theory can be expressed in the worldline formalism in terms of the quantum theory described in Polchinski's book.
 A: The faulty assumption is that the elements of the BRST cohomology can be represented by the elements of the unconstrained Hilbert space. They are really not, which means that one can't borrow the definition of the unconstrained inner product and use it. One has to invent another definition for the physical inner product that in this particular problem turns out to not be positively definite.
Let's consider something conceptually simpler first – the solutions of the $H \psi = 0$ constraint equation. After all you don't really need BRST in this problem if you're able to solve the quantum constraint $H \psi = 0$ constraint defined on the unconstrained Hilbert space.
One might naively assume that such elements belong to the unconstrained Hilbert space $\mathcal{H}$, but actually they do not. The reason is that zero lies in the continuum part of the spectrum of $H$ rather than being an isolated eigenvalue. This means that there aren't really any eigenvectors of $H$ in $\mathcal{H}$ with eigenvalue zero (a simple example of this is the operator $\hat{x}$ from 1d quantum mechanics – zero lies in its continuous spectrum, and so there doesn't exist a wavefunction that is its eigenfunction with eigenvalue zero – the delta function is not a function and is not square integrable).
Instead, the solutions of the quantum constraint live in the dual space $\mathcal{H}^{*}$. We say that $\varphi \in \mathcal{H}^{*}$ is a solution of the quantum constraint if
$$
\forall \psi \in \mathcal{H}: \quad \varphi \left( H \psi \right) = 0.
$$
Actually, I lied a bit there. To make everything more precise, one needs rigged Hilbert spaces. Then the space of which the solutions of the quantum constraint are elements is the dual of the dense subspace $S$ of rapidly decreasing wavefunctions that is part of the definition of the rigged Hilbert space.
Alternatively, you may try to solve the differential constraint equation which is just the KG equation:
$$
\left( \Box + m^2 \right) \psi(x^{\mu}) = 0.
$$
You will realize that the nonzero solutions of this equation are plane waves which are not elements of $\mathcal{H}$, because they don't have a finite norm (with the definition of norm coming from the spacetime integral that you have in your question).
It is in general nontrivial to solve the quantum constraint equation with zero in the continuous spectrum of the constraint operator. Several techniques exist such as group averaging, refined algebraic quantization, master constraint etc.
Fun fact (though not directly related to your question) – people in the LQG community have been trying to solve the quantum Hamiltonian constraint equation from General Relativity using these methods for decades. Some progress has been made including a few candidate models.
Now with BRST, since $Q = c H$ you have that the solutions of $H \psi = 0$ are also solutions of $Q \psi = 0$. It is not hard to convince yourself that the same caveat about dual spaces applies to the BRST cohomology when $0$ is in the continuous spectrum of $Q$. In fact, using BRST for this problem is a bit of an overkill, and the result of the application of BRST is just that the physical states are those that satisfy $H \psi = 0$.
Among other things it means that it is nontrivial to define the inner product on the space of solutions. You can't simply borrow the definition of the inner product on $\mathcal{H}$, because the solutions are not elements of $\mathcal{H}$. One has to demonstrate for an individual model that this is possible to define the physical inner product on solutions, it doesn't automatically follow from the definition of $Q$.
In the case of point particles, the physical inner product is given by the $3d$ integral from your question. It is not positive definite, which poses some problems for the interpretation of the theory (but not for the mathematics of the theory, it is still an interesting model and a prelude to string quantization).
