# Variational principle for canonical ensemble

In all textbooks I know, the derivation of the canonical probability distribution starts from the microcanonical ensemble. In my opinion, this is more of a motivation than a proper derivation, since some hypothesis must be introduced, which might not account for all possible conceivable physical systems.

Alternatively, the second law of thermodynamics states that the variation of entropy of a closed system is $$\Delta S \ge 0$$. If $$\Delta S \neq 0$$, then the entropy of the system is changing and, consequently, it is not in equilibrium. The only way to stop this is to attain its maximum, in which case the variation stops and equilibrium is attained. Consequently, an equivalent condition for equilibrium is that the state minimizes the free energy $$F = U -TS$$.

Consider a system with fixed volume $$\Lambda \subset \mathbb{R}^{d}$$, temperature $$T$$ and number of particles $$N$$. The configuration space is $$\Gamma_{\Lambda}:= (\Lambda\times \mathbb{R}^{d})^{N}$$, and the system is described by some Hamiltonian $$H_{\Lambda,\beta,N}$$, with $$\beta = 1/T$$ as usual. Since $$\Gamma_{\Lambda}$$ is a metric space, let it be equipped with its Borel $$\sigma$$-algebra $$\mathbb{B}_{\Lambda}$$ and let $$\mathcal{M}$$ be the set of all probability measures $$\mu$$ on $$(\Gamma_{\Lambda}, \mathbb{B}_{\Lambda})$$ which are absolutely continuous with respect to the Lebesgue measure on $$\mathbb{R}^{2dN}$$. In other words, every $$\mu \in \mathcal{M}$$ is of the form $$d\mu = \rho dx$$ for some positive real-valued measurable function $$\rho$$.

Given a measure $$\mu \in \mathcal{M}$$, the "number of accessible states" of the system is $$\mu(\Gamma_{\Lambda})$$, so it seems natural to define the entropy $$S_{\Lambda, \beta, N}$$ of this system by: $$S_{\Lambda,\beta,N}(\mu) := -k_{B}\ln \mu(\Gamma_{\Lambda})$$

My question is: Is this scenario correct and consistent with the canonical ensemble? And if so, is the canonical ensemble distribution $$d\mu_{\Lambda,\beta,N} = \frac{1}{Z_{\Lambda,\beta,N}}e^{-\beta H_{\Lambda,\beta,N}}dx$$ the solution of the following variational principle: $$\inf_{\mu \in \mathcal{M}}(\mathbb{E}_{\mu}[H_{\Lambda,\beta,N}] - TS_{\Lambda,\beta,N}(\mu)),$$ where $$\mathbb{E}_{\mu}[\cdot]$$ denotes the expectation with respect to the measure $$\mu$$?

Putting in another words, can we obtain the canonical distribution from a variational principle minimizing the free energy?

Let me discuss the simplest possible setting, in which the set of possible states $$\Omega$$ is finite. Think, for instance, of a finite-volume Ising model (or any other finite system, each of whose variables take only finitely many values).

Let $$\mu\in\mathcal{M}(\Omega)$$ be a probability measure on $$\Omega$$ and denote by $$H:\Omega\to\mathbb{R}$$ the energy. In this case, the entropy is naturally defined as $$S(\mu) = -k_B\sum_{\omega\in\Omega} \mu(\omega)\ln\mu(\omega).$$ (Note that you don't want to define it as $$S(\mu)=-k_B\ln\mu(\Omega)$$, which in any case would be equal to $$0$$ since $$\mu$$ is a probability measure. The fact that $$\Omega$$ represents all possible microstates does not make such a definition reasonable, since these microstates correspond in general to different values of the energy. The same is true of the definition you propose to use: the correct definition of the entropy would involve the relative entropy of $$\mu$$ with respect to the Lebesgue measure, $$S(\mu) = -k_B\int \mu(dx) \ln(d\mu/dx)$$, where $$d\mu/dx$$ denotes the Radon-Nykodim derivative.)

You then have $$\mathbb{E}_\mu[H] - T S(\mu) = \sum_{\omega\in\Omega} \bigl( H(\omega) + \frac{1}{\beta} \ln\mu(\omega) \bigr) \mu(\omega) = -\frac{1}{\beta}\sum_{\omega\in\Omega} \ln\Bigl[\frac{e^{-\beta H(\omega)}}{\mu(\omega)}\Bigr] \mu(\omega) .$$ By Jensen's inequality, $$\sum_{\omega\in\Omega} \ln\Bigl[\frac{e^{-\beta H(\omega)}}{\mu(\omega)} \Bigr]\mu(\omega) \leq \ln \biggl[ \sum_{\omega\in\Omega} \frac{e^{-\beta H(\omega)}}{\mu(\omega)} \mu(\omega) \biggr] = \ln \sum_{\omega\in\Omega} e^{-\beta H(\omega)} = \ln Z_\beta,$$ where $$Z_\beta = \sum_{\omega\in\Omega} e^{-\beta H(\omega)}$$ is the partition function. Moreover, the inequality becomes an equality if and only if the function $$e^{-\beta H(\omega)}/\mu(\omega)$$ is constant, that is, if and only if $$\mu(\omega) = \frac{1}{Z_\beta} e^{-\beta H(\omega)}.$$ We conclude that $$\inf_{\mu\in\mathcal{M}(\Omega)} \bigl(\mathbb{E}_\mu[H] - T S(\mu) \bigr)$$ is equal to the free energy $$F(\beta)=-kT\ln Z_\beta$$ and is reached exactly when $$\mu$$ is the Gibbs measure.

Some remarks:

• All the above extends to the case of particles in $$\Lambda\subset\mathbb{R}^d$$. See, for instance, §5.5 in Gallavotti's book.
• Things become much more interesting in the thermodynamic limit (that is, for an infinite system of particles, or of spins, say). Indeed, in that case, there is, in general, not a unique Gibbs measure. The point is that a version of the above still holds nevertheless, at least for translation-invariant Gibbs measures. This is discussed in many places, for instance in Section 6.9 of our book. You can actually go even further and show that translation-invariant Gibbs measures can be identified with the tangent functionals to the pressure, seen as a functional in the space of all interactions; see Chapter 16 in Georgii's book, for instance.
• This is an amazing answer! Thank you so much! I was not sure about my definition of entropy; I knew the discrete version, but I wasn't sure how to define it more generally when the configuration space is not finite. One thing that comes to my mind is that you did not mention that measures of $\mathcal{M}(\Omega)$ are absolutely continuous with respect to Lebesgue measure, but I believe this is necessary in the continuous case, am I write? Otherwise, how to ensure that the Radon-Nykodim derivative exists? Commented Aug 12, 2023 at 18:25
• One more thing: I just checked Gallavotti's book and it skips the proof of the variational principle. However, in the setting of finite volume, I repeated your reasoning with the sum $\sum_{\omega \in \Omega}$ replaced by the expectation $\mathbb{E}_{\mu}$ and $\mu(\omega)$ inside the logarithm replaced by the Radon-Nykodim derivative $d\mu/dx$ and it seems that I get to the very same conclusion. Am I missing something? (The fact that you decided to treat the finite case makes me wonder if there is something wrong in using the same approach in the continuous case). Commented Aug 12, 2023 at 19:08
• The argument should indeed be the same in the continuum. I chose the discrete setting because that's what looks most natural to me (it avoids all technicalities that are not really relevant for the problem at hand; for instance, it hides the reference measure, so that the entropy looks maybe more natural)... In fact, the vast majority of rigorous equilibrium statistical mechanics is done on lattice systems (even proving the existence of a liquid gas phase transition in a system of particles in the continuum with 2-body interactions is still an open problem!). Commented Aug 12, 2023 at 19:37
• I did not check what was done in Gallavotti's book (I only remembered that he did discuss the topic and checked the relevant section number). If you want a real proof with all the gory details (dealing with the most interesting case of infinite systems), there is this paper by Georgii. As you'll see there, if the measure $\mu$ has no density with respect to Lebesgue, the relative entropy will just be set to infinity (see (3.5) in the paper). Commented Aug 12, 2023 at 19:37
• Yvan, this discussion got me curious. Does the grand canonical distribution also arises from a variational problem like this? If so, the correct thermodynamic potential to use would be the same free energy plus a term proportional to N? Commented Aug 13, 2023 at 17:46