# Is it possible to describe Shannon Entropy in terms of Micro- and Macrostates?

I am trying to understand the connection between the entropy in Thermodynamics vs. entropy in Information Theory and I think a good way to clarify/illustrate it would be to find an analogy of microstates and macrostates within Shannon's definition of entropy. If thermodynamic entropy is based on the 'ignorance' of microstates given a macrostate and the information entropy is based on the 'ignorance' of the message to be received in a given channel, what would correspond to micro- and macrostates in this communication channel? I know these are terms defined in thermodynamics, but since the equations are more and less the same, it appears to me that there should be analogies in information theory and that could be useful to compare these two definitions/contexts of entropy.

EDIT: After some further reading I put up the following table, however I am not sure if I'm doing it right:


╔══════════════════════════╦═══════════════════════════╗
║    THERMODYNAMICS        ║     INFORMATION THEORY    ║
╠══════════════════════════╬═══════════════════════════╣
║ Phase-space              ║ Alphabet (sign set)       ║
║                          ║ + occurance               ║
╠══════════════════════════╬═══════════════════════════╣
║ Micro-states:            ║ Occurance of signs picked ║
║ location of particles    ║ from the alphabet         ║
║ in the phase-space       ║                           ║
╠══════════════════════════╬═══════════════════════════╣
║ Macro-states:            ║ Probability distribution  ║
║ Probability distribution ║ of symbols to be picked   ║
║ of particles to be found ║ from the alphabet         ║
║ in bin B                 ║ (or to occur)             ║
╠══════════════════════════╬═══════════════════════════╣
║ Bin in the distribution: ║ Bin in the distribution:  ║
║ Quantized phase-space    ║ Symbol in the alphabet    ║
║ location                 ║                           ║
╚══════════════════════════╩═══════════════════════════╝



This is one of my favourite topics in Physics at large, recently. The answer is contained in E. T. Jaynes's wonderful paper Information theory and statistical mechanics (1957).

To summarise it, call $X$ (whatever mathematical object it may be), the microstate. It is, of course, unknown. However, we do know a macrostate $M$, which is a vector of quantities which are functions of $X$: $M=F(X)$. The value of $M$ is known, and also the functional form of $F$. This means that we could guess a probability distribution $P(X|M)$.

The way Jaynes does it is indeed based on entropy. When we have no information about $X$, it is reasonable (from a completely information-theoretic point of view, no physics) to choose the distribution that has the largest possible Shannon entropy. But knowing the value of $M$ adds additional constraints, which appear in the form of Lagrange multipliers.

Long story short (I highly recommend reading the paper), the distribution you get is the same as the Canonical ensemble; Shannon entropy coincides with physical entropy; the temperature is the Lagrange multiplier associated to a given value of energy, and so on for other quantities (magnetic field and magnetisation...).

• So would it be correct to say the following? Micro-states: Location of particles in the phase-space vs. Occurance of signs picked from the alphabet Macro-states: Probability distribution of particles to be found in a quantized location of the phase-space vs. Probability distribution of symbols from the alphabet to occur
– ali
Commented May 6, 2017 at 18:29
• The macrostates are more the constraints on the distribution, for example total energy of the particles. But there's a correspondence. Commented May 6, 2017 at 18:38
• So each macrostate would be a bin of a histogram?
– ali
Commented May 6, 2017 at 18:54
• Or the histogram itself? Sorry I'm a little confused about the whole thing...
– ali
Commented May 6, 2017 at 19:08
• The mind game I like to thing about is imagining dividing up the microstates of a macrostate M into tiers (or resolutions?) consistent with my description/knowledge of M, such that each tier "covers all possibilities". If I do a good job, when I examine M after the fact to find out which microstates it is in, they will almost certainly be in the common microstates I assigned; and if they aren't, it means almost certainly my model sucks (or I cheated).
– Yakk
Commented Jul 28, 2022 at 18:01

I have found a comparison here between thermodynamic entropy and information which also provides some illustrations of macro- and microstates:

Particles in a box

In a simplified system with a 2-D box of particles, the macrostate is specified by energy E, number of particles N and volume V. There are a large number of possible microstates that are consistent with this system's single macrostate:

$S/k = log \Omega_p$

$\Omega_p = \text{number of equally probable microstates, k = Boltzmann's constant}$

Bits in a message

In a set of 4 x 4 array of bits (images/messages), the macrostate is specified by the number of bits (N = 16).

$H = log \Omega_p$

$\Omega_p = \text{number of equally probable messages}$

To summarize these illustrations, the macrostate would be the number of bits N, and the microstates would be all possible combinations of bits ("binary phase space").