Is there any intuitive reason behind why should the eigenfunctions of observables form a basis for our Hilbert space?

Question

Is there any intuitive reason behind why should the eigenfunctions of observables form a basis for our Hilbert space ?

For example, in the case of Stern-Gerlach experiment, sending the beam that has come out from the $+z$ direction to another SG apparatus, we can see that %100 percent of the beam is again in the $+z$ direction; this means that the state $|+z\rangle $ has no component in $|-z\rangle $, which intuitively explains the "orthogonal" part, but what about the existence of such a basis.

Answer 1

I think your question touches two different issues in quantum theory.

First, the mathematical one:

• is every operator such that its eigenfunctions are sufficient to form a basis of the Hilbert space of functions we work with?

I read an answer to this question in Slovak textbook Úvod do kvantovej mechaniky by Pišút, Gomolčák, Černý, 2nd edition, sec. 2.13. The relevant text is:

"It is assumed that systems of eigenfunctions pertaining to operators assigned to important physical quantities (energy, angular momentum etc.) form complete systems, i.e. it is possible to expand into series using eigenfunctions of such operator any arbitrary function $\Phi(x)$." $^1)$

$^1)$ "For some simple systems the statement can be proven. In general though the completeness of system of eigenfunctions has to be assumed or postulated."

The physical question I would also ask:

• When we consider all normalized eigenfunctions of some operator $\hat{A}$, do they define a Hilbert space that is big enough to explain all experiments, or are they sufficient only for some experiments, but a complete set of eigenfunctions is bigger or different and has to be yet found?

Here it may be that temporarily, the system of eigenfunctions defines a big enough Hilbert space, but later experiments may find some new degree of freedom, so it turns out that something was missing and the system of eigenfunctions was actually deficient (in the physical sense).

For example, consider spin. At first, Schroedinger was working with the Hilbert space that could be defined by obvious solutions of Schroedinger's equation with no spin. Later in 1927, Pauli modified the Hamiltonian to include spin which he already introduced in 1924 as a device to explain anomalous Zeeman effect. Now, when we take the old spinless Hamiltonian, its eigenfunctions are still useful, but by itself such system of eigenfunctions is not sufficient to explain that effect, so it isn't possible to expand every possible $\psi$ function in it. One has to introduce bigger Hilbert space, one which covers also the spin degree of freedom, and then we also have a new system of eigenfunctions.

Answer 2

If your system is in any state $|\psi\rangle$ and you measure an observable $A$, the probabilities to measure any outcome have to add up to 1.

If the eigenfunctions $\{|a_k\rangle\}$ of $A$ didn't form a basis, there would exist some $|\psi\rangle$ such that $$ \sum_k |\langle a_k | \psi\rangle|^2 < \langle\psi | \psi\rangle = 1 . $$ Here, the index "k" enumerates the different possible measurement outcomes and $P(k) = |\langle a_k | \psi \rangle|^2$ is the probability that the result of the measurement is "k".

Answer 3

Is there any intuitive reason behind why should the eigenfunctions of observables form a basis for our Hilbert space ?

The first thing to realize is that this is not so much a statement of what is true as a statement of what we would like to be true. It's aspirational. For example, the position operator $\hat{x}$ does not have any eigenvectors in the space of wavefunctions $\Psi(x)$. But Dirac wanted that to be true anyway, so he invented the delta function.

There have been some comments to the effect that this principle is just the spectral theorem, but I think the same example shows that that's not quite right either. The spectral theorem comes in various flavors, which have various hypotheses. For example, there is a version for bounded operators, meaning operators $\hat{A}$ for which there exists some constant $M$ such that we always have $|\hat{A}\Psi|\le M|\Psi|$. The $\hat{x}$ operator doesn't satisfy this requirement, nor do most of the operators we care about in physics.

So this really is a physical principle, not a mathematical theorem. One way of understanding the physical motivation is the following. Our psychological experience of doing measurements in quantum mechanics is well described by the Copenhagen interpretation, in which measuring $\hat{x}$ to have a value $x$ seems to result in the collapse of the wavefunction into a state of definite $x$. One consequence of this is that if we measure $\hat{x}$ twice in a row on the same system, we always get the same result. But if there was no eigenstate corresponding to $x$, then this would be impossible. After measuring the value $x$, the system would be in some state which was still a mixture of $x$ values, and then the second time around we would have some probability of getting some other $x$.

Another way of describing the physical motivation is that we would like systems with continuous degrees of freedom to act like systems with discrete and finite degrees of freedom. This corresponds to our classical intuition that there is no such thing as a measurement of a real number, only a measurement of things like integers or fractions that approximate a real number. Since the spectral theorem holds as a mathematical theorem for finite-dimensional vector spaces, we state it as an aspiration that we would like continuous systems to behave in the same way.

At the other extreme, we would also like, as a matter of mathematical convenience, to be able to talk about a basis that is not just infinite, as in the spectral theorem, but continuous, so that expressing a vector in our basis means integrating, not just doing an infinite sum. This is what happens in the example of the $\hat{x}$ operator, and also, for example, when we take a Gaussian pulse in position space and Fourier transform it into momentum space. The physical motivation is the same, but the mathematics is very different.