I have already parse other posts about this subject but none of them seems to answer completely to my interogation

[ this interrogation is to put in relation with this question Is there an equivalence between information, energy and matter? ]

indeed according to Bekenstein

the thermodynamic entropy and Shannon entropy are conceptually equivalent.

the number of arrangements that are counted by Boltzmann entropy reflects the amount of Shannon information that would be needed to implement any particular arrangement ...

...of matter and energy


for some, thermodynamic entropy can be seen as a specific instance of Shannon entropy. In short, the thermodynamic entropy is a Shannon entropy, but not necessarily vice versa.

for others, Shannon entropy is a mathematical quantity on ""abstract systems"" and have nothing to do with thermodynamics entropy.

so is there a consensual responce to that question:

Is there an equivalence between Boltzmann entropy and Shannon entropy..?

  • $\begingroup$ Just to be clear, you are asking if you can get $S=k_B \ln (W)$ from $S=-k\sum{p_i \ln(p_i)}$ correct? $\endgroup$
    – Jepsilon
    May 27 '18 at 7:28
  • $\begingroup$ @Jepsilon,... indeed the Boltzmann entropy is a special case of the Gibbs-Shannon entropy wich is the Shannon entropy formula applied to the microscopic states of a physical system so you are one of those who think that thermodynamic entropy can be seen as a specific instance of Shannon entropy...? $\endgroup$
    – alamata
    May 27 '18 at 8:08
  • 2
    $\begingroup$ There are many kinds of thermodynamic entropy, mainly microcanonical (Boltzmann), canonical (Gibbs), and grandcanonical (I think Landau was responsable for this one). All of which can be obtained from the Shannon entropy formula by setting the appropriate constraints. So in a way yes the Shannon entropy is a more general concept. It also generalises to quantities like Cross-entropy and KL-Divergence used to quantify similarities between distributions. $\endgroup$
    – Jepsilon
    May 27 '18 at 8:16
  • $\begingroup$ However like I stated in my answer, the Boltzmann entropy is a rather important special case because it gives a sharp upperbound on $S$ $\endgroup$
    – Jepsilon
    May 27 '18 at 8:19
  • 2
    $\begingroup$ @ Rococo, thx for the link,....if i have well understood, your pov is that thermodynamic entropy can be thought of as a particular application of information entropy. This pov meet Jepsilon's pov ... $\endgroup$
    – alamata
    Jun 11 '18 at 6:59

Boltzmann's entropy formula can be derived from the Shannon entropy formula when all states are equally probable.

Say you have $W$ microstates equiprobable with probability $p_i=1/W$. Then:

$S=-k\sum{p_i \ln p_i}=k\sum{ (\ln W)/W}=k\ln W$

Another way where this result can be obtained is maximising $S$ given that $\sum{p_i}=1$ using Lagrange multipliers:

$\max_{p_i}(S)= -k\sum{p_i \ln p_i} - \lambda(\sum{p_i}-1)$

Adding more constraints will result in a lower entropy distribution (such as the canonical entropy when adding the energy constraint and the grandcanonical when adding energy and particle constraints).

As a side note, it can also be shown that the Boltzmann entropy is an upperbound to the entropy that a system can have for a fixed number of microstates meaning:

$S\leq k \ln W$

This can also be interpreted as the uniform distribution being the distribution that provides the highest entropy (or least information, if you want someone was kind enough to prove this for me here https://math.stackexchange.com/questions/2748388/proving-that-shannon-entropy-is-maximal-for-the-uniform-distribution-using-conve).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.