Why is entropy a quantification of the typical fluctuation of internal energy around the expected value?

Question

In different books (one example is Statistical mechanics of learning, by Engel and Van Der Broeck) I stumbled upon an idea which should be elementary, but to me it is not easy to grasp. Entropy can be interpreted as the negative of the expected fluctuation of energy around its typical value. My own way to convince myself of this would be to write, in canonical ensemble, $$Prob(E)=\frac{e^{-\beta E}}{Z}=e^{-\beta\left(E-F\right)}=e^{-\beta\left(E-\left(\langle E\rangle-\frac{1}{\beta}\langle S \rangle\right)\right)}=e^{-\beta\left(E-\langle E\rangle-\left(-\langle \frac{1}{\beta}S\rangle\right)\right)}$$ from which one realises that the most probable value for $E-\langle E\rangle$ is $-\frac{1}{\beta}\langle S\rangle$. Is this explanation correct?

Answer 1

In the following, I'll set $k_B=1$. Usually, the fluctuations of energy are rather captured by heat capacity. This is captured by the following formula saying that heat capacity is proportional to the energy variance: $$ \begin{align} C &= \partial_T\langle E\rangle \\ &= \beta^2\Delta E^2 \end{align} $$ You can see this by noticing that $Z(\beta)$ is the moment generating function (you'll need to switch the sign for $\beta$ though), so $\ln Z$ is therefore the cumulant generating function (with the same sign caveat). The second order cumulant is just the variance $\Delta E^2$ and the first order cumulant the mean $\langle E\rangle$. The extra $\beta^2$ factor comes from the fact that in the definition of $C$, you take the derivative with respect to $T$, whereas in statistical physics, the natural variable is rather $\beta$.

I think that you were trying to get the large deviations, namely how the probability decays for energy different from the typical energy. The issue with your approach is that you are assuming that the energy is solely given by the Boltzmann factor. This is only the case for microstates. In general, a macrostate of definite energy has a large degeneracy which depends on energy and is given by entropy: $$ \begin{align} p(E) &= \frac{W(E)e^{-\beta E}}Z \\ &= \frac{e^{-\beta E+S_m(E)}}Z \\ &= e^{-\beta (F_m(E)-F_c)} \end{align} $$ where I used Boltzmann's formula for the microcanonical ensemble: $$ S_m = \ln W \\ F_m(E) = E-TS_m(E) $$ and the free energy of the canonical ensemble: $$ F_c = -T\ln Z $$ As you can see, the rate function which captures the fluctuations is not entropy, but rather ($\beta$ times) free energy. The heat capacity approach is just a quadratic approximation at its maximum. Notice that the free energy has an energy contribution and an entropic contribution. Typically, these two terms compete like in phase transitions where an ordered phase has a lower energy but lower entropy as well.

In the thermodynamic limit you usually have equivalence of the canonical and microcanonical ensembles. $F_c(\beta)$ is therefore the Legendre transform of $S_M(E)$. $F_m(E)$ is therefore minimised at $E=\langle E\rangle$, and $F_m(\langle E\rangle) = F_c(\beta)$. You also have $S_c = S_m(\langle E\rangle)$, i.e. the canonical entropy is the microcanonical entropy at the typical value of energy.

Take for example $N$ two level systems of energy $0,1$. The possible values for $E$ is $0,1,... N$ and the degeneracy is: $$ W(E) = \binom NE \\ S_m = \ln\binom NE \sim -N\left(\frac EN\ln\frac EN+\left(1-\frac EN\right)\ln\left(1-\frac EN\right)\right) $$ For the canonical ensemble, the energy distribution follows a binomial law: $$ \begin{align} P(E) &= \binom NE p^E(1-p)^{N-E} \\ p &= \frac1{1+e^\beta} & \beta &= \ln\frac{1-p}p \end{align} $$ You therefore identify: $$ Z = (1+e^{-\beta})^N \\ F_c = -NT\ln(1+e^{-\beta}) \\ \beta F_c = N\ln(1-p)\\ \langle E\rangle = pN = \frac N{1+e^\beta} \\ S_c = -N(p\ln p+(1-p)\ln(1-p))=N\left(\frac{\ln(1+e^\beta)}{1+e^\beta}+\frac{\ln(1+e^{-\beta})}{1+e^{-\beta}}\right) $$ The concentration of the energy at its mean value is given by the law of large numbers in the thermodynamic limit. The fluctuations are: $$ \Delta E^2 = \frac N{2(1+\cosh\beta)} $$ which have little to do with the entropy. In the thermodynamic limit, the fluctuations are governed by the Moivre-Laplace theorem (more generally the central limit theorem). The large deviations are given by $\beta F_m-\beta F_c$: $$ \ln P(E)\sim-N\left(\frac E N\ln\left(\frac {E/N}p\right)+\left(1-\frac E N\right)\ln\left(\frac {1-E/N}{1-p}\right)\right) $$

Hope this helps.