What's wrong with this derivation that $i\hbar = 0$?

Question

Let $\hat{x} = x$ and $\hat{p} = -i \hbar \frac {\partial} {\partial x}$ be the position and momentum operators, respectively, and $|\psi_p\rangle$ be the eigenfunction of $\hat{p}$ and therefore $$\hat{p} |\psi_p\rangle = p |\psi_p\rangle,$$ where $p$ is the eigenvalue of $\hat{p}$. Then, we have $$ [\hat{x},\hat{p}] = \hat{x} \hat{p} - \hat{p} \hat{x} = i \hbar.$$ From the above equation, denoting by $\langle\cdot\rangle$ an expectation value, we get, on the one hand $$\langle i\hbar\rangle = \langle\psi_p| i \hbar | \psi_p\rangle = i \hbar \langle \psi_p | \psi_p \rangle = i \hbar$$ and, on the other $$\langle [\hat{x},\hat{p}] \rangle = \langle\psi_p| (\hat{x}\hat{p} - \hat{p}\hat{x}) |\psi_p\rangle = \langle\psi_p|\hat{x} |\psi_p\rangle p - p\langle\psi_p|\hat{x} |\psi_p\rangle = 0$$ This suggests that $i \hbar = 0$. What went wrong?

EDIT: This seemingly little curiosity, as I perceived it initially, evolved into a major problem, as can be seen from the discussion below. Quantum mechanics indeed leads to non-physical results in my opinion and not just to resolvable paradoxes, at that regarding one of its basic claims, the one involving commutators. It seems most of what could be said on this issue has already been addressed below and probably there is no need to continue the exchange here. I would suggest that we discuss it further in the chat.

Answer 1

Both p and x operators as operators do not have eigenvectors in the strict sense. They have distributional eigenvectors which are only defined in a bigger space of functions than the space of square-normalizable wavefunctions, and which should be thought of as only meaningful when smeared a little bit by a smooth test function.

The normalization for $\langle \psi_p | \psi_p \rangle $ is infinite, because the p-wave is extended over all space. Similarly, the normalization of the delta-function wavefunction, the x-operator eigenvector, is infinite, because the square of a delta function has infinite integral.

You could state your paradox using $|x\rangle$ states too:

$$i\hbar \langle x|x\rangle = \langle x| (\hat{x}\hat{p} - \hat{p}\hat{x})|x\rangle = x \langle x|\hat{p}|x\rangle - \langle x|\hat{p}|x\rangle x = 0$$

because $|x'\rangle$ is only defined when it is smeared a little, you need to use a seprate variable for the two occurences of x'. So write the full matrix out for this case:

$$ i\hbar \langle x|y\rangle = x\langle x|\hat{p}|y\rangle - \langle x|\hat{p}|y\rangle y = (x-y)\langle x|\hat{p}|y\rangle$$

And now x and y are separate variables which can be smeared independently, as required. The p operator's matrix elements are the derivative of a delta function:

$$ \langle x|\hat{p}|y\rangle = -i\hbar \delta'(x-y)$$

So what you get is

$$ (x-y)\delta'(x-y)$$

And you are taking $x=y$ naively by setting the first factor to zero without noticing that the delta function factor is horribly singular, and the result is therefore ill defined without more careful evaluation. If you multiply by smooth test functions for x and y, to smear the answer out a little bit:

$$ \int f(x) g(y) (x-y) \delta'(x-y) dx dy= \int f(x)g(x) dx = \int f(x) g(y) \delta(x-y) $$

Where the first identify comes from integrating by parts in x, and setting to zero all terms that vanish under the evaluation of the delta function. The result is that

$$ (x-y)\delta'(x-y) = \delta(x-y)$$

And the result is not zero, it is in fact consistent with the commutation relation. This delta-function equation appears, with explanation, in the first mathematical chapter of Dirac's "The Principles of Quantum Mechanics".

It is unfortunate that formal manipulations with distributions lead to paradoxes so easiy. For a related but different paradox, consider the trace of $\hat{x}\hat{p}-\hat{p}\hat{x}$.

Answer 2

As it seems that you are not completely satisfied with Ron Maimon's answer, I'll put it in a bit different way.

The problem is that in your derivation you have a hidden ambiguity. $$\langle{\psi}_p\vert \hat{x}\vert\psi_p\rangle = \infty \;\;\;\;\; \Rightarrow \;\;\;\;\; \langle[\hat{x},\hat{p}]\rangle=...=(p-p)\langleψ_p|\hat{x}|ψ_p\rangle = 0\cdot\infty = \text{any number} $$ The problem is with the functions. Eigenfunctions of both momentum operator and coordinate operator are not really functions. They do not belong to integrable functions space and thus you can not freely work with them pretending that they are. Sometimes you can, but if you do at some point you find yourself in trouble.

If you take some "proper" function and do the math, you will find no problems. Let us take e.g. $$\psi(x)=\frac1{\sqrt{\pi}}e^{-x^2/2}$$ Then $$ \int_{-\infty}^{\infty} \psi(x) x\left( -i\hbar \frac{∂}{∂x} \right) \psi(x)dx = i\hbar \frac1{\pi} \int_{-\infty}^{\infty} x^2 e^{x^2}dx = i\hbar \frac1{2\sqrt{\pi}}$$ $$ \int_{-\infty}^{\infty} \psi(x) \left( -i\hbar \frac{∂}{∂x} \right)x \psi(x)dx = i\hbar \frac1{\pi} \int_{-\infty}^{\infty} \left(x^2-1\right) e^{x^2}dx = i\hbar \frac1{2\sqrt{\pi}}-i\hbar$$ The difference is what you expected.

If you take $\psi_a(x)=\frac1{a}\psi(x/a)$ and note that $\lim_{a\to0}\psi_a(x) = \delta(x) = \vert x\rangle$ you will get an idea how this paradox for $\vert x \rangle$ may be solved and check that the solution is a correct way to handle $0\cdot\infty$. Similar trick may be used to resolve your paradox. Just functions which have $\psi_p$ as a limit are less convinient.