Expected value of an operator in the microcanonical ensemble

Question

I am following professor David Tong's lecture notes on Statistical Mechanics and on page 9 of this file http://www.damtp.cam.ac.uk/user/tong/statphys/one.pdf he states that the expected value of an operator $\hat{\mathcal{O}}$ is $$ \langle \hat{\mathcal{O}} \rangle = \sum_{n} p(n) \langle n | \hat{\mathcal{O}} | n \rangle, $$ where $|n \rangle$ is the basis for the states which have the specified energy $E$ (in the microcanonical ensemble) and $p(n)$ is the probability of finding the system in such a state.

Well, given an arbitrary state $| \psi \rangle$, then $$ | \psi \rangle = \sum_{n} c_{n} |n \rangle, $$ where $p(n) = c_{n}^{*} c_{n} = |c_{n}|^{2}$. We also know from Quantum Mechanics that the expected value of an operator $\hat{\mathcal{O}}$ at a state $| \psi \rangle$ is given by $\langle \psi | \hat{\mathcal{O}} | \psi \rangle $. Using the expansion of $| \psi \rangle$ in the basis $|n \rangle$ we have $$ \langle \psi | \hat{\mathcal{O}} | \psi \rangle = \sum_{m,n} c_{m}^{*} c_{n} \langle m | \hat{\mathcal{O}} | n \rangle. $$ I used two different indices because there are two sums (one for the bra and another one for the ket).

Could someone explain the disagreement between the two equations?

Answer 1

Your claim that $p(n)$ is the $\lvert c_n \rvert^2$ from the decomposition $\lvert\psi\rangle = \sum_n c_n \lvert \psi_n \rangle$ is incorrect.

In statistical quantum mechanics, we must differentiate between a pure state of the system $\lvert \psi \rangle$ and a mixed state, which is given by a collection of states $\lvert \psi_n\rangle$ and the probability that the system is in that state $p_n$. The probabilities for the mixed state represent in some sense our uncertain knowledge about the system and do not mean that the system is thought to be in the quantum superposition of them, but are instead the quantum analogon of a given statistical "macrostate" to be in any one microstate. The $p_n$ are the probability that if we pick any system out of the statistical ensemble the mixed state describes, the system will be in the quantum state $\lvert \psi_n \rangle$, and this is not a probability for the result of a measurement as the $\lvert c_n \rvert^2$ of a pure state would be.

The expectation value for an observable $A$ is now given by the weighted average over its expectation values in the possible pure states, i.e $$ \langle A \rangle_\text{stat} = \sum_n p_n\langle A \rangle_{\psi_n}$$ where $\langle A \rangle_{\psi_n}$ is the usual expectation value $\langle\psi_n\vert A \vert \psi_n\rangle$, as it also would be in the classical treatment, where we would average over the value of the observable in the individual microstates of the ensemble weighted by their probabilities.

An equivalent way to encode this information and treat mixed and pure states within a unified formalism is to represent the statistical quantum state as a density matrix.