Inconsistency caused by the usage of matrix elements of the momentum operator in position representation

Question

Suppose $p$ is the momentum operator, we know that the matrix element of $p$ in the $q$-basis reads $\langle q'|p|q''\rangle=-i\hbar\frac{\partial}{\partial q'}\delta(q'-q'')=i\hbar\frac{\partial}{\partial q''}\delta(q'-q'')$.

Now consider the eigenequation $p|p'\rangle=p'|p'\rangle$. If we start with the LHS and insert two sets of completeness relations, we have $p|p'\rangle=\int dq'\int dq'' |q'\rangle\langle q'|p|q''\rangle\langle q''|p'\rangle=\int dq'\int dq'' |q'\rangle\left[i\hbar\frac{\partial}{\partial q''}\delta(q'-q'')\right]\langle q''|p'\rangle$.

We next integrate by part with respect to $q''$ to get

$p|p'\rangle=\int dq' |q'\rangle\left[i\hbar\delta(q'-q'')\langle q''|p'\rangle\right]|^{q''=+\infty}_{q''=-\infty}-\int dq'\int dq'' |q'\rangle\left[\frac{\partial}{\partial q''}\langle q''|p'\rangle\right][i\hbar\delta(q'-q'')]$ $=\left[i\hbar|q''\rangle\langle q''|p'\rangle\right]|^{q''=+\infty}_{q''=-\infty}+\int dq' |q'\rangle\left[-i\hbar\frac{\partial}{\partial q'}\langle q'|p'\rangle\right]$

$=\left[i\hbar|q''\rangle\langle q''|p'\rangle\right]|^{q''=+\infty}_{q''=-\infty}+\int dq' |q'\rangle\left[p'\langle q'|p'\rangle\right]$

$=\left[i\hbar|q''\rangle\langle q''|p'\rangle\right]|^{q''=+\infty}_{q''=-\infty}+p'|p'\rangle$.

Is there any mistake in the derivation that causes the appearance of the boundary term $\left[i\hbar|q''\rangle\langle q''|p'\rangle\right]|^{q''=+\infty}_{q''=-\infty}$?

---

Update: Let's look at the equation we obtained in the $p$-basis: $\langle p''|p|p'\rangle=\left[i\hbar\langle p''|q''\rangle\langle q''|p'\rangle\right]|^{q''=+\infty}_{q''=-\infty}+\langle p''|p'|p'\rangle$.

If $p''=p'$, we have $\langle p'|p|p'\rangle=\left[i\hbar\langle p'|q''\rangle\langle q''|p'\rangle\right]|^{q''=+\infty}_{q''=-\infty}+\langle p'|p'|p'\rangle$

$=\left(\frac{i}{2\pi}\right)|^{q''=+\infty}_{q''=-\infty}+\langle p'|p'|p'\rangle=0+\langle p'|p'|p'\rangle=\langle p'|p'|p'\rangle$. No problem.

If $p''\neq p'$, we have $\langle p''|p|p'\rangle=\left[\frac{i}{2\pi}e^{i\frac{q''(p'-p'')}{\hbar}}\right]|^{q''=+\infty}_{q''=-\infty}+\langle p''|p'|p'\rangle$.

To get a consistent result, we expect that $\left[\frac{i}{2\pi}e^{i\frac{q''(p'-p'')}{\hbar}}\right]|^{q''=+\infty}_{q''=-\infty}=0$. This sounds weird...

Nevertheless, I found the following argument in R. Shankar's book "Principles of Quantum Mechanics" (page 66):

The limit $\lim_{q''\to\infty}e^{i\frac{q''(p'-p'')}{\hbar}}$ should be defined to be "the average over a large interval":

$\lim_{q''\to\infty}e^{i\frac{q''(p'-p'')}{\hbar}}=\lim_{Q\to\infty,\Delta\to\infty}\frac{1}{\Delta}\int^{Q+\Delta}_Qdq''e^{i\frac{q''(p'-p'')}{\hbar}}=0$, if $p'\neq p''$.

If we choose to accept this, it seems that the inconsistency can be removed...

---

Another way to see that $\left[e^{iq''(p'-p'')/\hbar}\right]|^{q''=+\infty}_{q''=-\infty}=\left[e^{iq''(k'-k'')}\right]|^{q''=+\infty}_{q''=-\infty}=0$: $\left[e^{iq''(k'-k'')}\right]|^{q''=+\infty}_{q''=-\infty}=\int^\infty_{-\infty}d[q''(k'-k'')]e^{iq''(k'-k'')}=(k'-k'')\int^\infty_{-\infty}dq''e^{iq''(k'-k'')}=2\pi(k'-k'')\delta(k'-k'')=0$.

Answer 1

If we start with the LHS and insert two sets of completeness relations, we have $$p|p'\rangle=\int dq'\int dq'' |q'\rangle\langle q'|p|q''\rangle\langle q''|p'\rangle=\int dq'\int dq'' |q'\rangle\left[i\hbar\frac{\partial}{\partial q''}\delta(q'-q'')\right]\langle q''|p'\rangle$$

We already know that $\langle q''|p'\rangle = e^{ip'q''/\hbar}/\sqrt{2\pi\hbar}$, so we can make this somewhat more clear:

$$\hat p|p'\rangle = \frac{1}{\sqrt{2\pi\hbar}}\int dq' \int dq'' |q'\rangle\left[ i\hbar\frac{d}{dq''}\delta(q'-q'')\right]e^{ip'q''}$$

Integrating by parts yields

$$\frac{1}{\sqrt{2\pi\hbar}}\int dq' |q'\rangle \lim_{a\rightarrow-\infty}\lim_{b\rightarrow \infty} \left[i\hbar \delta(q'-q'')e^{ip'q''/\hbar}\right]_{q''=a}^{q''=b} - \frac{1}{\sqrt{2\pi\hbar}}\int dq' |q'\rangle(- p') e^{ip'q'/\hbar}$$

The first term is zero. The reason is that those limits are taken at fixed $q'$, and then the result is integrated over all $q'$. For any fixed $q'$, those limits evaluate to zero, and the integral of zero is zero. As expected,

$$\hat p|p'\rangle =p'\left( \frac{1}{\sqrt{2\pi\hbar}} \int dq' |q'\rangle e^{-ip'q'/\hbar}\right)= p'|p'\rangle$$

---

If you don't like that justification, note that the very definition of the (distributional) derivative of the delta function is the distribution $\delta'$ defined by

$$\int_{-\infty}^\infty \delta'(x-y) f(x) dx = -f'(y)$$

The heuristic "integration by parts" argument is ultimately a smokescreen; the apparently troublesome term is never actually there in the first place.