How is the expectation value defined in relativistic quantum mechanics? Since the norm of a wavefunction in relativistic quantum mechanics is defined as:
$$|\psi|^2=i\int\left(\psi^*\frac{\partial \psi}{\partial t}-\frac{\partial \psi^*}{\partial t}\psi\right)dx$$
How is the expectation value of an operator defined? I tried searching online but didn't find an answer.
 A: Yes, that is, in fact, the mathematically well defined norm of one-particle quantum states in relativistic  quantum mechanics (and relativistic QFT) for scalar Klein-Gordon particles.
However there is a substantial difference with respect to the non-relativistic case concerning the Hilbert space which does not contain all possible solutions of the scalar KG equation.
To appreciate this difference it is convenient to start form the momentum representation. Consider a sufficiently smooth real solution of KG equation. If it decays sufficiently fast in space can be expanded as (I henceforth  assume to deal in a 4-dimensional Minkowski spacetime referring to standard Minkowski coordinates with $c=\hbar=1$, $kx := \sum_{a=1}^3 k^ax^a$, $E(k):= \sqrt{k^2+m^2}$), then
$$\psi(t,x) = \int_{\mathbb{R}^3} \frac{\phi(k)}{\sqrt{2E(k)}} e^{i(kx - E(k)t)} + \frac{\overline{\phi(k)}}{\sqrt{2E(k)}} e^{-i(kx - E(k)t)} \:\:\frac{d^3k}{(2\pi)^{3/2}}\:.\tag{1}$$
The quantum state (at time $t$) associated to this solution is just one half of this decomposition:
$$\Psi_t(x):=  \int_{\mathbb{R}^3} \frac{\phi(k)}{\sqrt{2E(k)}} e^{i(kx - E(k)t)} \:\:\frac{d^3k}{(2\pi)^{3/2}}\:.\tag{2}$$
I stress that,
(a) differently from the real field $\psi$, $\Psi$ is complex in general;
(b)  $\Psi$ is still a solution of KG equation, but it cannot have spatial compact support;
(c)  in spite of the differences above, $\psi$ and $\Psi$ carry exactly the same amount of information.
As a matter of fact, with some elementary computations, one sees that
$$i\int_{\mathbb{R}^3} \overline{\Psi_t(x)} \frac{\Psi'_t(x)}{\partial t}
-  \Psi'_t(x) \frac{\overline{\Psi_t(x)}}{\partial t} d^3x = \int_{\mathbb{R}^3} \overline{\phi(k)}\phi'(k) d^3k\:.\tag{3}$$
Let us focus on the right-hand side.
(a) It is evidently positive-defined, so that the norm is positive as it should be;
(b) it is  also Poincaré-invariant in view of the structure of the unitary representation of the Lorentz group (the translational part acts trivially in terms of standard phases). If passing to the notation $K = (K^0,\vec{K})$ for the four momentum, so that $\vec{K}=k$ and $K^0:= E(k)$, and $\Lambda \in O(1,3)_+$,
$$(U_{\Lambda}\phi)(\vec{K}):= \sqrt{\frac{E(\vec{\Lambda K})}{E(\vec{K})}} \phi (\vec{\Lambda K})\:.$$
(Notice that $\frac{d\vec{K}}{E(\vec{K})}=\frac{d\vec{\Lambda K}}{E(\vec{\Lambda K})}$ as is well known and this fact assures the Poincaré invariance of the considered scalar product.)
(c) It does not depend on $t$.
Hence we have a properly defined Hilbert space (independent from $t$). Technically speaking one has to deal with a conveniently smooth vector space of functions $\psi$ which admits Fourier transform when divided with $E^{1/2}$ and finally he/she should take the completion of this space with respect to the said scalar product.
In particular, the expression for the scalar product given in the left-hand side of (3) is valid only for quantum states (2) and not for complete solutions of the KG equation as in (1).
REMARK. Relativistic QM has a problematic status in view of several issues in particular related to the definition of the position operators (there are several possibilities, but none is completely convincing and all are non-local). In particular $|\Psi_t(x)|^2$ cannot be interpreted as the probability density to find the particle at $x$ (when time is $t$).  Generally speaking only one-particle states whose energy content is smaller than the mass of the considered particle (thus photons are ruled out) have some chances to have some physically meaningful interpretation.
However, the Hilbert space constructed above has a deep relevance also in the standard approach to QFT. Indeed, the symmetric Fock space of QFT is exactly constructed upon this one-particle Hilbert space. Together with the vacuum state, that Hilbert space  is the crucial building block of the Fock space  construction.

Let us come to the issue regarding the expression of the expectation value of an observable. It is clear form the discussion above that, if we deal with the momentum representation, where the scalar product is
$$\langle \phi , \phi' \rangle := \int_{\mathbb{R}^3} \overline{\phi(k)} \phi'(k) d^3k$$
nothing relevant changes with respect to the standard formalism. The expectation value $<A>_\phi $, defined from the spectral theory for the selfadjoint operator $A$, satisfies the identity (if $\phi$ stays in the domain of the operator)
$$<A>_\phi = \int_{\mathbb{R}^3} \overline{\phi(k)} (A\phi)(k) d^3k\:.$$
To export this identity in the spacetime representation is a hard issue and it strictly depends on the nature of $A$. Abstractly speaking, from (2), the spacetime representation $A_{st}$ of $A$ is defined as
$$(A_{st}\Psi)_t(x):= \int_{\mathbb{R}^3} \frac{(A\phi)(k)}{\sqrt{2E(k)}} e^{i(kx - E(k)t)} \:\:\frac{d^3k}{(2\pi)^{3/2}}\:.$$
With this definition, the expectation value (provided a number of mathematical hypotheses are fulfilled) of $A$ in the spacetime reads from (3)
$$<A>_\phi = i\int_{\mathbb{R}^3} \left[\overline{\Psi_t(x)} \frac{(A_{st}\Psi)_t(x)}{\partial t}
-  (A_{st}\Psi)_t(x) \frac{\overline{\Psi_t(x)}}{\partial t}\right] d^3x\:.$$
To find the explicit expression of $A_{st}$ is usually  difficult, barring trivial cases, as the momentum.
A nice fact is that the Schroedinger equation with Hamiltonian $H:= \sqrt{k^2+m^2}$ in momentum representation,  when represented in  spacetime and for the considered states implies  the KG equation.
However, the spacetime representation of the said Hamiltonian is a pseudodifferential operator, that is a non-local operator
$$H_{st} = \sqrt{-\Delta_x + m^2I}$$
where $\Delta_x$ is the spatial Laplacian:
$$i\frac{\partial \Psi_t}{\partial t}=  \left(\sqrt{-\Delta_x + m^2I} \Psi_t\right)(x)$$
A: Actually a scalar product which is required on a Hilbert space cannot be defined by the expression:
$$\langle\psi,\phi\rangle = i \int\left(\psi^\ast \frac{\partial \phi}{\partial t} -  \phi \frac{\partial \psi^\ast}{\partial t}\right) dx $$
since for being a scalar product one requires that
$$\langle\psi,\psi\rangle \ge 0, $$
but this property is not fulfilled for the expression above on a space which includes all solutions of the Klein-Gordon equation, in particular the negative frequency solutions (it can be easily seen that it can take on negative values for instance for $\psi = Ae^{i(Et-px)}$). This is actually one of the problems the physicists encountered upon the construction of relativistic quantum theory. So it is not surprising that you did not find anything with "this product" since it does not exist.
It was soon realized that relativistic Quantum theory has to be a multi-particle theory, in particular because special relativity admits change from mass to energy and vice versa, therefore particles can change to other particles easily. The latter can only described in a multi-particle theory.
A multi-particle theory is defined on the so called Fock space where the states a described by particle content or differently said by the number of particles it contains and the states the particles take on. A typical Fock space state function is for instance a multi-particle state with 1 electron in a ground state and 1 photon which undergoes a transition to just 1 excited electron and 0 photons (this is just a simple example of the infinite possibilities):
$$|i\rangle =|1_{ph}, 1_{e_0}\rangle \quad \text{changes to} \quad  |f\rangle = | 0_{ph}, 1_e^\ast\rangle$$
where $|i\rangle $ means initial multi-particle state and $|f\rangle$ final multi-particle state. The transition describes the absorption of a photon by an electron in the (for instance) ground state that after the absorption is changed to an excited state.
$1_{e_0}$ means 1 one electron in the ground state and $1_e^\ast$ one electron in the excited state. An operator, for instance called $\hat{V}$, has to be written as an operator acting on Fock states
i.e. an operator which is developed in creation and annihilation operators. So if for instance
$$\hat{V} = \int \hat{j}^\mu \hat{A}_\mu d^3 x$$
and $\hat{A_\mu} = \sum_n (c^\dagger_n A^\ast_{n\,\mu}  +  c_n A_{n\,\mu})$
where $c^\dagger_n$ and $c_n$ are creation and annihilation operators which fulfill the rules:
$$\langle N^n_{ph} -1 | c_n | N^n_{ph}\rangle = \langle N^n_{ph} | c^\dagger_n | N^n_{ph}-1\rangle  =\sqrt{N^n_{ph}}$$
where $N_{ph}^n$ is the number of photons in the state $n$. Then the matrix element
$$V_{fi} = \langle f|\hat{V} |i\rangle =  \langle 0_{ph}, 1_e^\ast |\hat{V} | 1_{ph}, 1_{e_0} \rangle = \int \langle  1_e^\ast | \hat{j}^\mu|1_{e_0} \rangle  \langle  0_{ph} | \hat{A}_\mu |  1_{ph}\rangle  $$
makes sense and can be calculated. Actually, the formalism of Fock space is rather vast and cannot be explained here in detail. It is just to give you an idea how matrix elements in relativistic Quantum theory (or even Quantum Field Theory) have to be computed.
There exist also semi-relativistic approaches which -- if they  make sense in the regime where they are applied -- after some "tricks" still use the formalism of one-particle wave functions, the formalism which is used in non-relativistic Quantum mechanics.
On the other hand, the Fock state formalism has turned out so useful that it is also often applied to multi-particle problems in non-relativistic Quantum mechanics, in particular in condensed matter physics.
