# Why is the free energy minimized by the Boltzmann distribution?

Can someone show me, without glossing over anything, why $F = E - TS$ is minimized when $p_i = e^{-U_i/k_bT}/\sum_ie^{-U_i/k_bT}$? I understand it conceptually, but am having difficulty showing it formally.

-
Where are you having trouble? –  Michael Brown Feb 22 '13 at 1:15
I know that $E-TS=\sum_iU_ip_i+k_BT*\sum_ip_iln(p_i)$ I take the derivative of F with respect to $p_i$, yielding $\sum_iU_i+k_BT*\sum_i(1/p_i)$, and substitute the Boltzmann distribution for $p_i$ expecting to get 0, but I'm having trouble with the algebra. Is this a reasonable approach? –  user21243 Feb 22 '13 at 1:19
@user21243 - It would be a good idea to post what you've tried along with your question. –  Kitchi Feb 22 '13 at 5:24

You are on the right track.

Let me begin by being super explicit about what we are trying to prove for the benefit of all readers.

We assume, for the sake of simplicity (both mathematical and conceptual) that the system in question is finite-dimensional so that the probability distribution that describes its statistical mechanical state can be represented as a finite sequence of $p_i$'s that sum to one; \begin{align} \rho = (p_1, \dots, p_n), \qquad \sum_{i=1}^n p_i = 1. \end{align} Next, we define a function $F$ as follows \begin{align} F(\rho, T) = E(\rho) - TS(\rho) \end{align} where the ensemble average energy $E$ and entropy $S$ are defined as \begin{align} E(\rho) = \sum_{i=1}^n p_iU_i, \qquad S(\rho) = -k_B \sum_{i=1}^np_i\ln p_i. \end{align} for a given distribution $\rho$, and for a given value of $T$. For notational convenience, be denote the Boltzmann probability distribution corresponding to temperature $T$ by \begin{align} \rho^B(T) = (p_1^B(T), \dots, p_n^B(T)), \qquad p_i^B(T) = \frac{e^{-U_i/k_BT}}{\sum_i e^{-U_i/k_BT}} \end{align} Then I claim that

1. The distribution $\rho^B(T)$ is a critical point of $F(\rho, T)$ in the space of all possible probability distributions $\rho$.
2. The distribution $\rho^B(T)$ is a global minimum of $F(\rho, T)$ in the space of probability distributions whose energy matches that of the $\rho^B(T)$.

Proof. To prove statement 1, that $\rho^B(T)$ is a critical point of $F(\rho, T)$, we note that the claim we want to prove is a constrained optimization problem because we want to verify that $F$ has a critical point among all probability distributions, which are described by special sequences of $p_i$'s, namely those that sum to one. We therefore can't just take derivatives of $F$ with respect to $p_i$ and call it a day; we need to enforce the constraint that changes in $F$ that we consider preserve the property that the sum of the probabilities is one.

The standard way to do this is to introduce a Lagrange multiplier to enforce the desired constraint, but let's do this in a more direct, intuitive way. Suppose that we want to perturb the distribution $\rho = (p_1, \dots, p_n)$, but make sure that this perturbation doesn't affect the fact that the probabilities sum to 1. If we write our perturbations as $p_i +\delta p_i$, then this means that \begin{align} 1 = \sum_{i=1}^n (p_i + \delta p_i) = \sum_{i=1}^n p_i + \sum_{i=1}^n \delta p_i \end{align} which, gives \begin{align} \sum_{i=1}^n \delta p_i = 0. \tag{$\star$} \end{align} This condition ensures that we never stray away from probability distributions when we do these perturbations.

Next, let's examine what happens to $F$ to first order in $\delta p_i$ when we perturb the distribution $\rho$ in this way. The first order change in $F$ is given by \begin{align} \delta F(\rho, T) &=\sum_{i=1}^n\frac{\partial F}{\partial p_i}(\rho, T)\delta p_i\\ &= \sum_{i,j=1}^n \frac{\partial}{\partial p_i}(p_jU_j -k_BTp_j\ln p_j)\delta p_i \\ &= \sum_{i,j=1}^n (\delta_{ij}U_j + k_B T (\delta_{ij}\ln p_j - \frac{p_j}{p_j}\delta_{ij}))\delta p_i \\ &= \sum_i(U_i + k_BT\ln p_i - k_BT)\delta p_i. \end{align} Now here is where the magic happens. Let us evaluate this first order change on the Boltzmann distribution at temperature $T$; \begin{align} \delta F(\rho^B(T), T) &= \sum_i(U_i + k_BT(-\frac{U_i}{k_BT} - \ln Z) - k_BT)\delta p_i\\ &= \sum_{i=1}^n(U_i - U_i - k_BT(\ln Z +1))\delta p_i \\ &= -k_BT(\ln Z +1)\sum_{i=1}^n\delta p_i \\ &= 0 \end{align} where in the last step, we used the constraint $(\star)$. This is exactly the desired result! The first order change around the Boltzmann distribution vanishes which is precisely the statement that this distribution is a critical point of $F$.

To prove statement 2, we consider $F(\rho)$ for any $\rho$ whose ensemble average energy is equal to that of the Boltzmann distribution; \begin{align} E(\rho) = E(\rho^B(T)). \end{align} If we compute the difference on the free energies of $\rho$ and $\rho^B(T)$, we find that \begin{align} F(\rho, T) - F(\rho^B(T)) &= E(\rho) - T S(\rho) - E(\rho^B(T)) + TS(\rho^B(T)) \\ &= T\Big(S(\rho^B(T))-S(\rho)\Big) \end{align} Now, recall that the Boltzmann distribution is precisely that which maximizes the entropy among all distributions that share the same ensemble average energy $E$, in other words \begin{align} S(\rho^B(T)) \geq S(\rho). \end{align} It follows immediately that \begin{align} F(\rho^B(T))\leq F(\rho) \end{align} which is exactly the desired result; the free energy for any other distribution with the same ensemble average energy is greater than that of the Boltzmann distribution!

-
There's some errors with +/- signs in the second and third equations after the 'magic happens', otherwise looks good! –  Mark A May 1 '14 at 4:18
@MarkA Thanks for the careful read! Leave it to me to totally botch signs. In any case, they should be fixed now. Let me know if you happen to find any more errors. –  joshphysics May 1 '14 at 4:26