In introductory courses on Statistical Mechanics, much is derived from using the generalized entropy of some system (such as the canonical partition function etc):
$$\tilde{S} = k_B \sum_{n}p_n\ln(p_n)$$ However, not much insight is usually given into why this is entropy is the case (such as how we can see the correspondence between Entropy in Thermodynamics) which would appear to be useful. How can one either derive or come to an appreciation of this equation?

  • 1
    $\begingroup$ Do you mean the proving uniqueness ( up to the constant) of Shannon entropy subject to some additive and montonicity assumptions? $\endgroup$ – AHusain Sep 22 '16 at 6:31
  • $\begingroup$ @AHusain I'm sure most reasonable readers would take that to be one of many excellent and intellectually fulfilling motivations for the use of Shannon's notion, which, IMO, is as good as it gets in physics. A concise, readable account of this approach is section 2 in E. T. Jaynes, "Information Theory and Statistical Mechanics", Phys. Rev. 1957 $\endgroup$ – WetSavannaAnimal Sep 22 '16 at 7:41

Perhaps one way to see the correspondence to entropy in thermodynamics is as follows. First note that there is usually a minus sign in the definition of entropy (otherwise, entropy will be negative).

Now consider the Boltzmann distribution $p(\omega) = Z^{-1} e^{-\beta H(\omega)/k_B}$ (suppose we are working on a finite state space). The entropy of $p$ is \begin{align} \tilde S &= -k_B \sum_\omega p(\omega) \log p(\omega) \\ &= -k_B \sum_\omega Z^{-1} e^{-\beta H(\omega)/k_B}(-\beta H(\omega)/k_B - \log Z)\\ &= \beta \langle H \rangle + k_B \log Z. \end{align} Re-arranging this gives \begin{equation} -\beta^{-1} k_B \log Z = \langle H \rangle - \beta^{-1} \tilde S, \end{equation} relating free energy ($-\beta^{-1} k_B \log Z$), internal energy ($\langle H \rangle$), and entropy.

This argument may appear circular depending on how one derives the Boltzmann distribution, but nevertheless hopefully provides some reassuring consistency.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.