This question is based on a footnote in Michael Talagrand's "What is a Quantum Field Theory?".
To understand the footnote, we need a few definitions: $X_m$ is the "mass shell," i.e., $X_m = \{ \, (\omega, k) \in \mathbb R^{1,3} \, | \, \omega^2 = k^2 + m^2 \, \}$.$$X_m = \{ \, (\omega, k) \in \mathbb R^{1,3} \, | \, \omega^2 = k^2 + m^2 \, \}.$$ $U$ is a specific representation of the Poincaré group acting on $L^2(X_m, \lambda_m),$ where $\lambda_m$ is the Poincaré-invariant measure on $X_m$. Namely, for $(a, A) \in \mathcal P$, we have $U(a, A)(\varphi)(p)=e^{i\langle a, p\rangle} \varphi(A^{-1}p)$.$$U(a, A)(\varphi)(p)=e^{i\langle a, p\rangle} \varphi(A^{-1}p).$$
The footnote 42 on page 125 reads:
One can view (in a sense) $U(a,A)$ as operating on the space of functions on $\mathbb R^{1,3}$ whose Fourier transform is supported by $X_m$. These functions can be characterized by a certain differential equation, or “wave equation”, the Klein-Gordon equation which will play an important role later. This approach is very popular in physics. My problem here is that it is not clear what is the norm which makes the representation unitary. I never ran into a single textbook describing such a norm, or even mentioning the need for it, and I am unable to explain what is the mental picture physicists form at this point. More generally more complicated representations of the Poincaré group can be defined on more complicated spaces of functions satisfying certain wave equations.
So, what is the norm (in fact, inner product) that Talagrand wants but doesn't know? Actually, we need a little more than just an inner product: We need a rigorous definition of the space under consideration. Talagrand refers to "the space of functions on $\mathbb R^{1,3}$ whose Fourier transform is supported by $X_m$," but I think this needs more qualifications to make sense: Are we talking about Schwarz functions, or some other space of well-behaved functions? General $L^2$ functions are probably too broad, because $X_m$ is a set of measure zero in $\mathbb R^{1,3}$. Intuitively, the plane waves satisfying the KG equation correspond to delta functions on $X_m$, but we need to put this on rigorous footing and define an honest Hilbert space, and I'm not sure how this can be done.