I am trying to understand the connection between the entropy in Thermodynamics vs. entropy in Information Theory and I think a good way to clarify/illustrate it would be to find an analogy of microstates and macrostates within Shannon's definition of entropy. If thermodynamic entropy is based on the 'ignorance' of microstates given a macrostate and the information entropy is based on the 'ignorance' of the message to be received in a given channel, what would correspond to micro- and macrostates in this communication channel? I know these are terms defined in thermodynamics, but since the equations are more and less the same, it appears to me that there should be analogies in information theory and that could be useful to compare these two definitions/contexts of entropy.

EDIT: After some further reading I put up the following table, however I am not sure if I'm doing it right:

║ Phase-space              ║ Alphabet (sign set)       ║
║                          ║ + occurance               ║
║ Micro-states:            ║ Occurance of signs picked ║
║ location of particles    ║ from the alphabet         ║
║ in the phase-space       ║                           ║
║ Macro-states:            ║ Probability distribution  ║
║ Probability distribution ║ of symbols to be picked   ║
║ of particles to be found ║ from the alphabet         ║
║ in bin B                 ║ (or to occur)             ║
║ Bin in the distribution: ║ Bin in the distribution:  ║
║ Quantized phase-space    ║ Symbol in the alphabet    ║
║ location                 ║                           ║


This is one of my favourite topics in Physics at large, recently. The answer is contained in E. T. Jaynes's wonderful paper Information theory and statistical mechanics (1957).

To summarise it, call $X$ (whatever mathematical object it may be), the microstate. It is, of course, unknown. However, we do know a macrostate $M$, which is a vector of quantities which are functions of $X$: $M=F(X)$. The value of $M$ is known, and also the functional form of $F$. This means that we could guess a probability distribution $P(X|M)$.

The way Jaynes does it is indeed based on entropy. When we have no information about $X$, it is reasonable (from a completely information-theoretic point of view, no physics) to choose the distribution that has the largest possible Shannon entropy. But knowing the value of $M$ adds additional constraints, which appear in the form of Lagrange multipliers.

Long story short (I highly recommend reading the paper), the distribution you get is the same as the Canonical ensemble; Shannon entropy coincides with physical entropy; the temperature is the Lagrange multiplier associated to a given value of energy, and so on for other quantities (magnetic field and magnetisation...).

| cite | improve this answer | |
  • $\begingroup$ So would it be correct to say the following? Micro-states: Location of particles in the phase-space vs. Occurance of signs picked from the alphabet Macro-states: Probability distribution of particles to be found in a quantized location of the phase-space vs. Probability distribution of symbols from the alphabet to occur $\endgroup$ – ali May 6 '17 at 18:29
  • 1
    $\begingroup$ The macrostates are more the constraints on the distribution, for example total energy of the particles. But there's a correspondence. $\endgroup$ – Bzazz May 6 '17 at 18:38
  • $\begingroup$ So each macrostate would be a bin of a histogram? $\endgroup$ – ali May 6 '17 at 18:54
  • $\begingroup$ Or the histogram itself? Sorry I'm a little confused about the whole thing... $\endgroup$ – ali May 6 '17 at 19:08

I have found a comparison here between thermodynamic entropy and information which also provides some illustrations of macro- and microstates:

Particles in a box

enter Microcanonical ensemble, the set of all possible configurations of particles in a box

In a simplified system with a 2-D box of particles, the macrostate is specified by energy E, number of particles N and volume V. There are a large number of possible microstates that are consistent with this system's single macrostate:

$S/k = log \Omega_p$

$\Omega_p = \text{number of equally probable microstates, k = Boltzmann's constant}$

Bits in a message

enter Binary phase space, all possible combinations of bits

In a set of 4 x 4 array of bits (images/messages), the macrostate is specified by the number of bits (N = 16).

$H = log \Omega_p$

$\Omega_p = \text{number of equally probable messages}$

To summarize these illustrations, the macrostate would be the number of bits N, and the microstates would be all possible combinations of bits ("binary phase space").

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.