Quantum superposition in density matrix formalism

Question

I was thinking about quantum superposition and stumbled into something that made me quite uncomfortable. Consider a qubit with Hamiltonian eigenstates $|0\rangle$ and $|1\rangle$. To each of these eigenstates we can associate a corresponding density operator;

$$\rho_0=|0\rangle\langle0|,\quad\rho_1=|1\rangle\langle1|.$$

If we then write the most general pure state in the Bloch coordinates we have

$$|\psi\rangle=\cos\frac{\theta}{2}|0\rangle+e^{i\phi}\sin\frac{\theta}{2}|1\rangle,$$

which corresponds to the density matrix

$$\rho=|\psi\rangle\langle\psi|=\cos^2\frac{\theta}{2}|0\rangle\langle0|+e^{i\phi}\sin\frac{\theta}{2}\cos\frac{\theta}{2}|1\rangle\langle0|+e^{-i\phi}\sin\frac{\theta}{2}\cos\frac{\theta}{2}|0\rangle\langle1|+\sin^2\frac{\theta}{2}|1\rangle\langle1|.$$

This clearly cannot be written as a superposition of $\rho_0$ and $\rho_1$. There's a pair of elements ($|0\rangle\langle1|$ and $|1\rangle\langle0|$) which is not generated when we take the eigenstates as a basis. This bothers me since it seems to imply quantum superpositions are not quite like addition in the underlying space. The cross terms seem to show up out of nowhere. By applying the definitions, the terms are there and the calculations work out. But can someone give me some intuition as to why quantum superpositions have that form instead of a simpler, more intuitive

$$\rho=r\rho_0+(1-r)\rho_1?\quad(r\in[0,1])$$

Answer 1

The off-diagonal terms (which come from taking the outer product of a non-basis vector with itself to form a pure density matrix) are exactly the mathematically precise form of what we mean when we say "quantum coherence". They are the secret sauce that makes quantum mechanics truly different from ordinary classical probabilities, which take the exact form that you give in your last equation. As such, they are arguably the core of quantum mechanics. Indeed, the process of "decoherence", which roughly means "the process by which quantum superpositions turn into ordinary classical probabilities", mathematically means precisely the loss of these off-diagonal terms in the density matrix which are bothering you.

Here are three slight variations on your questions, and my answers to each:

1. What is the fundamental "reason" for these off-diagonal terms? What do they "really mean"?

Philosphers have been hotly debating this question for decades, and there doesn't seem to be any end in sight.

2. How can I develop intuition for the physical consequences of these weird off-diagonal terms?

In light of the above discussion, this is basically equivalent to the question "how can I develop intuition for how quantum mechanics differs from classical probability theory?" Quantum mechanics is notoriously unintuitive, and the answer varies very much from person to person. Some people like the "hydrodynamic" analogy in which the wavefunction is like water waves with destructive interference, etc., but your mileage may vary.

3. What are the precise physical and mathematical consequences of these off-diagonal terms? Can they lead to anything which can't also come from classical probability distributions (i.e. convex combinations of orthonormal pure states)?

In my opinion, the entire subject of quantum information theory is dedicated to answering this question. It's an enormous subject on which entire books have been written, which I couldn't possibly do justice here. But the place to start is the two core results which arguably really launched the field: the Kochen-Specker theorem and Bell's theorem, which were (confusing) both discovered by Bell. These results were the first to show that quantum mechanics really is a fundamentally different theory from classical probability theory, rather than just a different formalism.