How to derive Shannon Entropy from Clausius Theorem?

Question

I am studying Quantum Information now, and I need to understand the entropy of a quantum system. But before I go there, I need to understand Shannon Entropy which is defined as :

$H(X) = -\sum_{i=1}^{n} {p(x_i) \log_2{p(x_i)}} $

where $X$ is a discrete random variable with possible outcomes $x_{1},...,x_{n}$ which occur with probability $p_{1},...,p_{n}$. This is entropy that works in information theory, but we know that entropy is already defined back way in thermodynamics by Clausius as :

$$dS = \frac{\delta Q}{T}$$

Then, in statistical physics, entropy is defined by Boltzmann as :

$S=k_B \ln{\Omega}$

where $\Omega$ is the number of microstates of a system. How can I derive the Shannon entropy from these thermodynamics and statistical physics entropy definitions?

Answer 1

These are not the same.

Shannon entropy (Information entropy), $H_\alpha=-\sum_i p_i\log_\alpha p_i$ applies to any system with specified probabilities $p_i$.

Boltzmann entropy, defined via the famous $S=k\log\Omega$ implies that the system occupies all the accessible states with equal probability, $p_i=1/\Omega$ (this is a particular case of the Information entropy, as can be seen by plugging $p_i=1/\Omega$ into the Shannon formula, taking natural logarithm base, and discarding customary dimensional coefficient).

Gibbs entropy, defined via the Clausius inequality, $dS\geq \delta Q/T_{env}$, is defined empirically, as a quantity that always monotonuously increases, and thus makes thermodynamic processes irreversible.

Furthermore, Boltzmann entropy and Gibbs entropy can be shown to be equivalent, reflecting the equivalence between the microscopic statistical physics and the phenomenological thermodynamics.

Finally, Let me first point out that entropy may mean different things. As Jaynes, in his article The minimum entropy production principle claims that there are six different types of entropy with somewhat different meaning.

Remark:
There is some disagreement about what is called Gibbs entropy, as Gibbs actually introduced two - one along the lines of Clausius, and another more similar to Boltzmann entropy. These are sometimes referred to as Gibbs I and Gibbs II. For more ways to introduce entropy see this answer to Is information entropy the same as thermodynamic entropy?.

Answer 2

Yes, you can.

I'm going to show that the expression of the Shannon entropy can be deduced from statistical and thermodynamic relations.

We know that the entropy defined by $\mathrm{d}S=\delta{Q}/T$ can be related to thermodynamic potentials. The formula we will use here is $$ F=U-TS, $$ where $U$ is the energy and $F$ is the Helmholtz function (sometimes maybe called the free engergy?).

And in statistical mechanics, $F$ is related to the partition function $Q$ for the canonical ensemble: $$ F=-kT\ln Q, $$ where $Q=\sum_{r}\mathrm{e}^{-\beta E_r}$ is the partition function (of course you know that $\beta = \frac{1}{kT}$).

The probability of any microstate $r$ is given by $$ P_r=\frac{\mathrm{e}^{-\beta E_r}}{Q}. $$

OK. Here it comes. Take its logarithm and average it, and we have Eq. (3.3.13) in (Pathria & Beale 2021) $$ \langle \ln P_r \rangle=-\ln Q-\beta \langle E_r\rangle=\beta(F-U) = -\frac{S}{k}. $$ The average formula is just $$ \langle \ln P_r \rangle = \sum_{r} P_r \ln P_r. $$ At last, we obtain the Shannon entropy (forget about the Boltzmann constant $k$) $$ S = -\sum_{r} P_r \ln P_r. $$

---

Ref.

R. K. Pathria, Paul D. Beale. Statistical Mechanics 4th ed., Academic Press, 2021.

Answer 3

You can't. It is not possible to derive a more general formula from a less general one. Of course, one can find hints for the generalization, but the validity of the generalization has to be proved independently.

The relationship between the formulas is the following: Shannon's formula is more general (it applies to every probability distribution, even non-equilibrium ones and even if there is no energy underlying the probabilities).

Statistical mechanics entropies (different in different ensembles) are the special case of Shannon's formula for the case where the probabilities have the correct values for each ensemble.

Clausius formula has a connection with statistical mechanics entropies only at the so-called thermodynamic limit. I.e. in the limit of a very large system.

Answer 4

A better approach would be to use the Shannon Entropy to derive Gibbs entropy: $S=−k\cdot∑p_n \cdot \ln(p_n)$. The two equations are very similar and therefore it is much easier understand. From there it is easy to arrive at Boltzman entropy and finally Clausius.

Answer 5

You can derive the Gibbs entropy, which is simply the Shannon entropy multiplied by the Boltzmann constant, from the Boltzmann entropy and a multinomial distribution. This video proceeds with the derivation:

https://www.youtube.com/watch?v=h3xVAVcYfjk

The Boltzmann entropy is not a particular case of the Gibbs entropy in the demonstration. It's quite the other way around. Sure, the Boltzmann entropy has equal probabilities while they may differ in the Gibbs entropy. However, the probabilities refer to microstates of a canonical ensemble in the Boltzmann entropy, but to individual particle energy states in the Gibbs entropy.