Why do different ways of computing $\langle p^2 \rangle$ require integration by parts to match?

Question

I'm learning the basics of quantum mechanics from Binney and Skinner's book, and I'm trying to do a very basic exercise (2.5d), yet am struggling. The exercise is to calculate $\langle p^2 \rangle$. Super straightforward stuff. Here's why I'm a bit confused: if you apply the definition of the expectation value, you get that

\begin{align*} \langle p^2 \rangle = \langle \psi | p^2 | \psi \rangle. \end{align*}

Going a bit further, I wrote down the following calculations.

\begin{align*} \langle \psi | \hat{p} \circ \hat{p} | \psi \rangle & = \langle \psi | \hat{p} \left( \int dx \ |x \rangle \langle x | \hat{p} | \psi \rangle \right)\\\\ & = \langle \psi | \hat{p} \left( \int dx \ |x \rangle \left( -i \hbar \frac{\partial \psi}{\partial x} \right) \right)\\\\ & = - i \hbar \langle \psi | \hat{p} \left( \int dx \ |x \rangle \frac{\partial \psi}{\partial x} \right)\\\\ & = - i \hbar \left( \int dx \ \langle \psi | \hat{p} | x \rangle \frac{\partial \psi}{\partial x} \right)\\\\ & = - i \hbar \left( \int dx \ (\langle x | \hat{p} | \psi \rangle)^* \frac{\partial \psi}{\partial x} \right)\\\\ & = \hbar^2 \int dx \ \left( \frac{\partial \psi}{\partial x} \right)^* \frac{\partial \psi}{\partial x}. \end{align*} If you integrate by parts on the final answer, you can recover the "usual" answer of \begin{align*} \langle p^2 \rangle = - \hbar^2 \int dx \ \psi^* \frac{\partial^2 \psi}{\partial x^2}. \end{align*} My question is this: why do these two methods yield "different" initial answers? Is there anything deeper to this, or is it just a mathematical necessity?

Answer 1

You are more or less assuming $\hat p$ is self-adjoint so that $$ \langle \psi\vert \hat p^2\vert\psi\rangle= \langle \psi\vert\hat p\hat p\vert\psi\rangle= \langle \psi\vert \hat p^\dagger \hat p\vert\psi\rangle = \langle \hat p \psi\vert\hat p\psi\rangle $$ and you technically need integration by parts to move one derivative in $$ \int dx \psi^*(x)\frac{d}{dx}\frac{d}{dx}\psi(x) $$ to the left, plus the assumption that the boundary terms disappear (usually $\psi(\pm\infty)=0$) and “smooth enough” functions $\psi(x)$. Using Dirac notation: $$ \langle \psi\vert \hat p^\dagger \sim \langle \hat p\psi\vert $$ so that $\psi(x)^*\frac{d}{dx}\sim \left(\frac{d}{dx}\psi(x)\right)^*$ using integration by parts, since the derivative is initially on the right of $\psi^*(x)$.

Doing this will produce something that goes like $$ \langle\hat p\psi\vert\hat p\psi\rangle=\int dx \left(\frac{d}{dx} \psi(x)\right)^* \left(\frac{d}{dx}\psi(x)\right) $$ when you use coordinates. Note that I (purposely) left out $i$’s and $-i$’s: you can chase those by yourself. Finally, the assumptions on $\psi(x)$ are essential here, as are the boundary conditions: see this question or this question for technicalities when the boundary conditions used above are not valid.

Answer 2

I am not sure why you chose to apply it that way. I mean, in particular, why did you not attempt $$ \begin{align} \tag1\left<\hat p{}^2\right> &=\int\left<\psi|x\right>\mathrm dx\left<x|\hat p{}^2|\psi\right> \end {align} $$ so that then you will not require the integration by parts? Anyway, the IbP is a proof of equivalence and so technically you can say that the symmetric version is also the $\hat p{}^2$ if you want.

Answer 3

They aren't different - by utilizing integration by parts, you are explicitly demonstrating that your final and penultimate expressions are equal to one another.