# Why is (von Neumann) entropy maximized for an ensemble in thermal equilibrium?

Consider a quantum system in thermal equilibrium with a heat bath. In determining the density operator of the system, the usual procedure is to maximize the von Neumann entropy subject to the constraint that the ensemble average of the Hamiltonian has a fixed value.

What justifies this assumption?

Sakurai, in his QM text, writes

To justify this assumption would involve us in a delicate discussion of how equilibrium is established as a result of interactions with the environment.

I'd appreciate if someone could shed some light on this. References are welcome as well.

I've heard the suggestion that a thermal equilibrium ensemble is simply defined by that density operator which solves the constrained optimization problem above. If this is the case, then why are real physical systems that are in weak contact with heat baths for long periods well described by such mathematical ensembles, and how would one identify the Lagrange multiplier $\beta$ that arises with inverse temperature of the heat bath?

You need to read this paper by Jaynes. I can't explain it as well as him, but I will try to summarise the main points below.

The first thing is to realise that the entropy is observer-dependent: it depends on what information you have access to about the system. A finite temperature means that you don't have access to all the information about the state of the system; in particular, you cannot keep track of the (infinite) degrees of freedom of the bath. However, suppose that some demon could keep track of all the degrees of freedom of the system and bath: he/she sees zero entropy. For the demon, it looks a bit like the total system is at zero temperature (although really it is better to say that temperature is ill-defined for the demon).

Given that you are ignorant (sorry, but at least I'm not calling you a demon), you need to find a consistent prescription for assigning probabilities to the different microstates. The prescription must be 'honest' about what you do or don't know. The entropy is in some sense a unique measure of ignorance, as proved by Shannon. Therefore you should 'maximise your ignorance', subject to the constraint that you do know certain macroscopic observables, e.g. average energy or average particle number if the system is open, etc.

Maximising the entropy of the system is the most logical way to assign probabilities to the microstates of the system, given access only to a limited subset of observables. The same 'MaxEnt' principle is quite general and applies to all statistical analysis, not only physics. The Lagrange multiplier $\beta$ is identified with inverse temperature by comparing the outcome of this abstract procedure to the experimental facts of phenomenological thermodynamics.

If you are interested in the actual dynamics of equilibration, there has been a lot of literature on this recently, especially in mesoscopic systems. Particular focus is laid on the integrability of the system: non-integrable (chaotic) systems do thermalise, whereas there is a fair bit of evidence that integrable systems do not thermalise properly. Intuitively, this is because integrable systems have a maximal set of locally conserved quantities so that, even when in contact with a heat bath, the memory of the initial conditions is never quite lost.

See, for example: Dynamics of thermalisation in small Hubbard-model systems and Thermalization and ergodicity in many-body open quantum systems, if you search 'thermalization' (sic) on arxiv then you will find many more.

• Thanks Mark. Yeah I actually forgot about the information-theoretic viewpoint for some reason when I was asking this question. I guess I was hoping that the MaxEnt principle could somehow be justified not only from the perspective of statistical inference, but that it would emerge in equilibrium ensembles as a consequence of the process of equilibration (or something to that effect). Do you think such an expectation is unreasonable? I'm not really sure what to think on this point. Also, thanks very much for the references at the end. Feb 6, 2013 at 19:22
• I think this is a very good question, and it's the subject of a lot of research at the moment, although it comes with a caveat. In the context of classical physics, it's important to remember that the system is always really in a pure state. So the emergence of the Gibbs ensemble cannot really be separated from the observer's subjective knowledge about the system. I think of it as ignorance/information/entropy being created dynamically each time the system interacts with degrees of freedom that the observer cannot track. Feb 6, 2013 at 19:30
• (contd.) When the energy/particle etc. distribution is uniform over scales longer than the minimal achievable experimental resolution then further interactions do not increase the observer's entropy, i.e. the probability distribution hits the steady state. Feb 6, 2013 at 19:34
• (contd.) However, in quantum mechanics you can't say the system is always really in a pure state, and the uncertainty of the observer is not subjective, it's required by the laws of physics. You can study equilibration of small quantum systems in a heat bath and you find that the system evolves towards a thermal state, basically due to entanglement with the bath. Likewise, it has been shown that almost all bipartite states for systems with a natural system-environment division are locally thermal, see here. Feb 6, 2013 at 19:39
• Wow thank you so much Mark. I couldn't have asked for better responses; I'll definitely look at all of the references you suggested. Feb 6, 2013 at 19:43

Here is an alternate approach to answering this question (which ignores temperature and lagrange multipliers) given to us by the 1st law of thermodynamics and quantum information.

In short, a weakly interacting ensemble at thermal equilibrium maximises the von Neumann entropy because in doing so it minimises the free energy of the system.

Why can we say this? A weakly interacting ensemble at thermal equilibrium corresponds to a Gibbs state, which is nothing more than a quantum version of a canonical ensemble. We can write this out as $$\hat{\rho}_\beta = \frac{e^{-\beta \hat{H}}}{\mathcal{Z}}$$ where $$\mathcal{Z}$$ is the canonical partition function. Canonical ensembles also have a related quantity known as the Gibbs Free energy given by $$F = U - tS$$ where $$U$$ is the internal energy and $$t$$ is the temperature. We can write out a quantum version of this as $$F = \langle \hat{H} \rangle - tS.$$ For a Gibbs state, this will coincide with the Gibbs Free energy but what if we define this quantity for an arbitrary quantum state modeling an open quantum system taking $$S$$ to be the von Neumann entropy $$S = -\text{tr}\{\hat{\rho}\ln\hat{\rho}\}.$$ In general we have $$F(\hat{\rho}) = \text{tr}\{\hat{\rho}\hat{H}\} + \beta^{-1}\text{tr}\{\hat{\rho}\ln\hat{\rho}\} = \beta^{-1}\text{tr}\{\hat{\rho}(\ln\hat{\rho}+\beta\hat{H})\}$$ by linearity of the trace. The Free energy of a Gibbs state is the Gibbs Free energy $$F(\hat{\rho_\beta}) = -\beta \ln \mathcal{Z} = -\beta^{-1}\ln\left(\text{tr}\{e^{-\beta \hat{H}}\}\right).$$ Considering the relative entropy of this arbitrary state $$\hat{\rho}$$ and the Gibbs state $$\hat{\rho}_\beta$$ we have $$D(\hat{\rho}||\hat{\rho}_\beta) = \text{tr}\{\hat{\rho}\ln\hat{\rho}\} - \text{tr}\{\hat{\rho}\ln\hat{\rho}_\beta\}$$ which can be expressed in terms of the Free Energy as $$D(\hat{\rho}||\hat{\rho}_\beta) = \beta \left(F(\hat{\rho}) - F(\hat{\rho}_\beta)\right).$$ But, the relative entropy is known to be nonnegative $$D(\cdot||\cdot)\geq 0.$$ As a result the Gibbs Free energy is the lowest possible free energy or the free energy of the Gibbs state is minimal.

The first law of thermodynamics can be stated in this context as $$dE = tdS +dF$$ so if Gibbs states minimise the free energy, they maximise the von Neumann entropy, which happens to be the Gibbs Entropy.

Beyond this one can also justify the approach to thermal equilibrium on the grounds of quantum information expanding on this argument. Any quantum channel that preserves the Gibbs state cannot increase the free energy. Rather, the free energy of out-of-equilibrium states is monotonically decreasing to the Gibbs Free Energy giving equilibrium, maximimising von Neumann entropy on long times.

This argument is given by Preskill in his notes on Quantum Shannon Theory.