What's the correct link between Dirac notation and wave mechanics integrals?

Question

In wave mechanics when we compute the expectation value of energy we write the following

$$\left<\hat{H}\right>=\int_{-\infty}^\infty\mathrm{d}x\ \psi^*(x)\hat{H}\psi(x)=\int_{-\infty}^\infty\mathrm{d}x\ \psi^*(x)E\psi(x)$$

In Dirac notation it is simply written as

$$\left<\hat{H}\right>=\left<\psi\left|\hat{H}\right|\psi\right>$$

We then chooses some orthogonal basis $\left|x\right>$ and $\left|x'\right>$ in the position space and expand the dirac notation above as follows (with the limits ($-\infty$ to $\infty$) omitted for simplicity)

$$\left<\hat{H}\right>=\left<\psi\left|\hat{H}\right|\psi\right>$$ $$=\int \mathrm{d}x'\int \mathrm{d}x\left<\psi\left|x'\right>\left<x'\right|\hat{H}\left|x\right>\left<x\right|\psi\right>$$ Now since the basis is orthogonal and $\hat{H}$ has eigenvalues $E$ thus the following equation is obeyed

$$\left<x'\left|\hat{H}\right|x\right>=\left<x'\left|E\right|x\right>\tag{1}$$

Since $E$ is just a constant, it can be taken out from the brackets

$$\left<x'\right|E\left|x\right>=E\left<x'\right|\left. x\right>$$ and since the bases are orthogonal $$\left<x'\right |\left. x\right>=\delta(x'-x)\tag{2}$$ Thus the expectation value integral becomes $$=\int\mathrm{d}x\int\mathrm{d}x\left<\psi\left|x'\right>E\delta(x'-x)\left<x\right|\psi\right>$$ Integrating with respect to $x'$ $$=\int\mathrm{d}x\left<\psi\left|x\right>E\left<x\right|\psi\right>=\int\mathrm{d}x\ \psi^*(x)E\psi(x)$$

1. Are the steps in $(1)$ and $(2)$ legal?

2. If question 1 is true, is it legal to do a similar treatment for other operators e.g. $\hat{p}$, $\hat{a}^\dagger$, $\hat{j}$ etc. to recover their wave mechanics counterparts of the expectation value from the dirac notation?

3. If question 1 is false, how to correctly (and preferably mathematically rigorously) recover the wave mechanics result of the expectation value of any operator $\hat{A}$ (not necessary self adjoint/hermitian) from the Dirac notation $\left<\psi\right|\hat{A}\left|\psi\right>$?

Answer 1

Your step $[1]$ is not OK. Just because $\hat H$ has eigenvectors whose eigenvalue is $E$ does not mean that any vector will be an eigenvector of $\hat H$ with eigenvalue $E$. In particular, the position eigenstates are emphatically not eigenstates of the hamiltonian.

To go beyond $$⟨\hat{H}⟩ = \int dx'\int dx⟨\psi|x'⟩⟨x'|\hat{H}|x⟩⟨x|\psi⟩$$ you need to specify further what you mean by $\hat H$ - if all you know is that it is some operator on $\mathcal H$ then you can't go any further, because there's no need for it to have any special relation to the position basis.

Usually, however, you are dealing with a non-relativistic single-particle hamiltonian of the form $H= p^2/2m +V(x)$. In this case, the correct way to handle $\hat H$ is to commute it with the bra to its left: $$ ⟨x'|\hat H=\left[-\frac{\hbar^2}{2m}\frac{\partial^2}{\partial x'^2} +V(x')\right]⟨x'|. $$ Here the derivative acts on anything $x'$-dependent to the right of the derivative. (To see more about this, try this thread.)

To implement this in your expression for $⟨\hat H⟩$, you first start by completing the sandwich: \begin{align} ⟨x'|\hat H|x⟩ &= \left[-\frac{\hbar^2}{2m}\frac{\partial^2}{\partial x'^2} +V(x')\right]⟨x'|x⟩ \\ &= \left[-\frac{\hbar^2}{2m}\frac{\partial^2}{\partial x'^2} +V(x')\right]\delta(x-x'), \end{align} since the position states are orthogonal. You then perform the $x$ integral, to get \begin{align} \int dx⟨x'|\hat{H}|x⟩⟨x|\psi⟩ &= \int dx\left[-\frac{\hbar^2}{2m}\frac{\partial^2}{\partial x'^2} +V(x')\right]\delta(x-x')⟨x|\psi⟩ \\&= \left[-\frac{\hbar^2}{2m}\frac{\partial^2}{\partial x'^2} +V(x')\right]⟨x'|\psi⟩. \end{align} You can then do the integral over $x'$, to get \begin{align} ⟨\hat H⟩=\int dx'\int dx⟨\psi|x'⟩⟨x'|\hat{H}|x⟩⟨x|\psi⟩ &= \int dx'⟨\psi|x'⟩\left[-\frac{\hbar^2}{2m}\frac{\partial^2}{\partial x'^2} +V(x')\right]⟨x'|\psi⟩ \\&= \int dx\psi(x)^*\left[-\frac{\hbar^2}{2m}\frac{\partial^2}{\partial x^2} +V(x)\right]\psi(x), \end{align} where I've relabelled $x'$ as $x$. This is the final result and there's nothing you can do to simplify it without further assumptions.

---

Your notation, $$\int_{-\infty}^\infty\mathrm{d}x\ \psi(x)^*E\psi(x) \tag{$*$}$$ is ambiguous and probably incorrect, but this depends on the context.

• If $E$ is a shorthand for the operator $-\tfrac{\hbar^2}{2m}\tfrac{\partial^2}{\partial x^2} +V(x)$, then this is formally correct but it is a misleading way to represent this. This is why we use hamiltonians and carets to denote operators, so that we keep clear tabs on what is a number and what is an operator.

• If $E$ is a $c$-number (i.e. not an operator) then the expression has very limited validity. More specifically, it is only valid if $\psi$ is an eigenstate of $\hat H$ with eigenvalue $E$, in which case you simply write $\hat H|\psi⟩=E|\psi⟩$ and therefore $⟨\hat H⟩=⟨\psi|\hat H|\psi⟩=E$.

There isn't really any case where the notation $(*)$ is both correct, useful, and non-misleading.

Answer 2

It seems to me that you are making some confusion. The problem with the passage [1] (and [2]) that you outline is that you are not allowed to do that (on a rigorous level) if the operator has continuous spectrum, for there are no corresponding eigenvectors on the Hilbert space (and it is wrong also on a non-rigorous level as pointed out by others). Anyways, Dirac notation is nothing fancy (in my opinion), just a convenient way to write scalar products in Hilbert spaces (someone may say there are Gel'fand triples and so on, but they do not add, again in my opinion, any relevant further insight in this particular context).

It is actually perfectly legitimate to look for the "wave mechanics notation" (i.e. on some $L^2$ space) starting from an abstract Hilbert space. When you write $\langle\psi, H\psi\rangle$, you are writing the scalar product on the space $\mathscr{H}$ between $H\psi$, where $H$ is a linear operator (supposedly densely defined, and $\psi$ is in its domain of definition), and $\psi$.

Now every separable Hilbert space (almost any Hilbert space utilized in quantum theories is separable) is isometrically isomorphic to some $L^2(\Omega,d\mu)$ (actually to $l^2$), even if the isomorphism can be difficult.

So, let's suppose that $\mathscr{H}$ is separable and you know explicitly the isometric isomorphism $i:\mathscr{H}\to L^2(\mathbb{R}^d)$, for some $d\in \mathbb{N}^*$. Let's say that $i\phi_n=\phi_n(x)\in L^2(\mathbb{R}^d)$, where $\{\phi_n\}_{n\in\mathbb{N}}\subset \mathscr{H}$ is an orthonormal basis in $\mathscr{H}$, and $\{\phi_n(x)\}_{n\in\mathbb{N}}\subset L^2$ is one on the latter. Then you have that $$\langle\psi, H\psi\rangle_{\mathscr{H}}= \langle i\psi, (iHi^{-1}) i\psi\rangle_{L^2}=\langle \psi(x), E(x,\partial_x) \psi(x)\rangle_{L^2}\; ,$$ where $\psi(x)=\sum_{n}a_n\phi_n(x)$ (if $\psi=\sum_n a_n \phi_n$), and $E(x,\partial_x)$ is a linear differential operator that corresponds to the operator $H$.

Now the form of $E(x,\partial_x)$ may not be easy, but if the isomorphism $i$ is known explicitly, then it could be recovered from $E(x,\partial_x)=iHi^{-1}$. Also, observe that in general $E(x,\partial_x)$ would be a differential operator, not just a function of $x$: a classical example is the operator $-\partial_x^2 + V(x)$, where $V$ is a suitable function.

Answer 3

While $[2]$ is "legal", $[1]$ definitely isn't. We are able to write: $$\int dx\ \psi^\ast H\psi = \int dx\ \psi^\ast E\psi = E$$ only if $H\psi = E\psi$ i.e. when $\psi$ is an eigenfunction of $H$. In the general case, it is not, and we generally have $\psi = \sum_n c_n \psi_n$, where $\psi_n$ are the eigenstates of $H$, with $H\psi_n = E_n\psi_n$.

The expression in the dirac notation is: $$\langle H \rangle = \langle\psi\rvert H\lvert\psi\rangle$$ $$ = \int dx' \int dx \langle\psi\vert x'\rangle\langle x'\rvert H\lvert x\rangle\langle x\vert\psi\rangle$$

Now, the $\lvert x\rangle$ are certainly not eigenfunctions of $H$, and you can't carry on with the replacement you proposed. The $\langle x'\rvert H\lvert x\rangle$ is called the 'matrix element' of $H$ between $x'$ and $x$.

What normally happens is that $H = H(x, p)$, and you know the matrix elements of $x$ and $p$ in the 'position representation' i.e. $\langle x'' \rvert x\lvert x'\rangle = x'\delta(x'' - x')$ and $\langle x''\rvert p\lvert x'\rangle = -i\hbar\frac{\partial}{\partial x''}\delta(x''-x')$ (also see this post). Based on these, together with matrix multiplication, you get the matrix elements of $H$.

For an eigenstate, it happens that, since $$H\lvert\psi\rangle = E\lvert\psi\rangle$$ $$\implies \int dx' \langle x''\rvert H\lvert x'\rangle\langle x'\vert\psi\rangle = E\langle x''\vert\psi\rangle$$ and you therefore get $$ \int dx' \int dx \langle\psi\vert x'\rangle\langle x'\rvert H\lvert x\rangle\langle x\vert\psi\rangle = \int dx' \psi^\ast(x')E\psi(x')$$ which is the expression in wave mechanics.

In the more general case, however, we have to work by replacing $H(x, p)$ with $H(x, -i\hbar\frac{\partial}{\partial x})$ to get an integral of the form: $$\int dx\ \psi^\ast(x)H(x, -i\hbar\frac{\partial}{\partial x})\psi(x)$$ This is in essence the operator formalism.