# How to derive Shannon Entropy from Clausius Theorem?

I am studying Quantum Information now, and I need to understand the entropy of a quantum system. But before I go there, I need to understand Shannon Entropy which is defined as :

$$H(X) = -\sum_{i=1}^{n} {p(x_i) \log_2{p(x_i)}}$$

where $$X$$ is a discrete random variable with possible outcomes $$x_{1},...,x_{n}$$ which occur with probability $$p_{1},...,p_{n}$$. This is entropy that works in information theory, but we know that entropy is already defined back way in thermodynamics by Clausius as :

$$dS = \frac{\delta Q}{T}$$

Then, in statistical physics, entropy is defined by Boltzmann as :

$$S=k_B \ln{\Omega}$$

where $$\Omega$$ is the number of microstates of a system. How can I derive the Shannon entropy from these thermodynamics and statistical physics entropy definitions?

• May 20 at 8:59
• What you may want is the careful derivation of Boltzmann's $H$ Theorem, which was what showed that there was a connection between entropy and (schematically) $p\log p$.
– Buzz
May 20 at 20:38

These are not the same.

Shannon entropy (Information entropy), $$H_\alpha=-\sum_i p_i\log_\alpha p_i$$ applies to any system with specified probabilities $$p_i$$.

Boltzmann entropy, defined via the famous $$S=k\log\Omega$$ implies that the system occupies all the accessible states with equal probability, $$p_i=1/\Omega$$ (this is a particular case of the Information entropy, as can be seen by plugging $$p_i=1/\Omega$$ into the Shannon formula, taking natural logarithm base, and discarding customary dimensional coefficient).

Gibbs entropy, defined via the Clausius inequality, $$dS\geq \delta Q/T_{env}$$, is defined empirically, as a quantity that always monotonuously increases, and thus makes thermodynamic processes irreversible.

Furthermore, Boltzmann entropy and Gibbs entropy can be shown to be equivalent, reflecting the equivalence between the microscopic statistical physics and the phenomenological thermodynamics.

Finally, Let me first point out that entropy may mean different things. As Jaynes, in his article The minimum entropy production principle claims that there are six different types of entropy with somewhat different meaning.

You can't. It is not possible to derive a more general formula from a less general one. Of course, one can find hints for the generalization, but the validity of the generalization has to be proved independently.

The relationship between the formulas is the following: Shannon's formula is more general (it applies to every probability distribution, even non-equilibrium ones and even if there is no energy underlying the probabilities).

Statistical mechanics entropies (different in different ensembles) are the special case of Shannon's formula for the case where the probabilities have the correct values for each ensemble.

Clausius formula has a connection with statistical mechanics entropies only at the so-called thermodynamic limit. I.e. in the limit of a very large system.

• It is not a more general formula, it is a formula for different thing. Shannon entropy, or information-theoretic entropy, is a different concept altogether from the Clausius entropy. Clausius entropy can be sometimes calculated as value of the Shannon entropy; but this does not make the latter a more general formula for the same thing. May 23 at 19:33
• @JánLalinský I do not agree with your point of view. Shannon entropy can be defined for every probability distribution, not only the one corresponding to an equilibrium ensemble. But if used with the equilibrium probability distribution of a Statistical Mechanics ensemble it provides exactly the Statistical Mechanics expression for the entropy. I would call this situation a particularization of a more general formula to a more specific case. In a similar way, the Statistical Mechanics entropy, in general, does not coincide with the thermodynamics entropy. However, with additional conditions... May 23 at 21:57
• ... i.e. with the thermodynamic limit, it coincides with the thermodynamic entropy. What else, to be allowed to speak about a transition from more general to more specific concepts? May 23 at 21:59
• > "...Statistical Mechanics...I would call this situation a particularization of a more general formula to a more specific case." -- that is true, but that generalization consists of going from Shannon entropy of special probability distribution to Shannon entropy of any probability distribution. Not from going from Clausius entropy to Shannon entropy. Clausius entropy isn't logically dependent on statistical mechanics or probability distributions. May 23 at 22:07
• @JánLalinský Sure, as Clausius himself, one can have Clausius entropy even without knowing that there is something called Statistical Mechanics. However, if Statistical Mechanics plus some additional prescription (existence of the thermodynamic limit) allows for justifying the properties of Clausius entropy, I would call it a more general theory. But maybe we are just disagreeing about the term generalization. May 23 at 22:25

Yes, you can. Talk is cheap. I'll show you the formula.

I'm going to show that the expression of the Shannon entropy can de deduced from statistical and thermodynamical relations.

We know that the entropy defined by $$\mathrm{d}S=\delta{Q}/T$$ can be related to thermodynamic potentials. The formula we will use here is $$F=U-TS,$$ where $$U$$ is the energy and $$F$$ is the Helmholtz function (sometimes maybe called the free engergy?).

And in statistical mechanics, $$F$$ is related to the partition function $$Q$$ for the canonical ensemble: $$F=-kT\ln Q,$$ where $$Q=\sum_{r}\mathrm{e}^{-\beta E_r}$$ is the partition function (of course you know that $$\beta = \frac{1}{kT}$$).

The probability of any microstate $$r$$ is given by $$P_r=\frac{\mathrm{e}^{-\beta E_r}}{Q}.$$

OK. Here it comes. Take its logarithm and average it, and we have Eq. (3.3.13) in (Pathria & Beale 2021) $$\langle \ln P_r \rangle=-\ln Q-\beta \langle E_r\rangle=\beta(F-U) = -\frac{S}{k}.$$ The average formula is just $$\langle \ln P_r \rangle = \sum_{r} P_r \ln P_r.$$ At last, we obtain the Shannon entropy (forget about the Boltzmann constant $$k$$) $$S = -\sum_{r} P_r \ln P_r.$$

Ref.

R. K. Pathria, Paul D. Beale. Statistical Mechanics 4th ed., Academic Press, 2021.

A better approach would be to use the Shannon Entropy to derive Gibbs entropy: $$S=−k\cdot∑p_n \cdot \ln(p_n)$$. The two equations are very similar and therefore it is much easier understand. From there it is easy to arrive at Boltzman entropy and finally Clausius.