Entropy has two definitions, which come from two different branches of science: thermodynamics and information theory. Yet, they both are thought to agree. Is it true?

Entropy, as seen from information theory, measures our ignorance about a system. Entropy, as seen from thermodynamics, measures the amount of extractable work. One is epistemological, the other is ontological. Yet, they both can agree, if entropy really measures the amount of extractable work, given one's knowledge about the system.

But, is this true? Does Shannon's entropy computed on the probability distribution of a physical system express the amount of work we can obtain from it? How do increases in our knowledge increase the amount of extractable work? Is Landauer principle powerful enough to establish the connection?

  • $\begingroup$ I find the dramatically different answers to this question a bit baffling. I also thought the two kinds of entropy were different, but I liked Joe's answer and upvoted it. Would it be possible for someone to reconcile the different answers, or point out why one (or more) of them is wrong? $\endgroup$
    – Aaron Sterling
    Commented Oct 5, 2011 at 18:55
  • 2
    $\begingroup$ One reconciliation is that "work" has multiple meanings. Typically the first meaning learned is thermodynamic, but (as Dirac demonstrated) a more general meaning amounts to: Given a class of dynamical processes, 'work' is any mathematically natural measure of what those processes accomplish $\endgroup$
    – John Sidles
    Commented Oct 5, 2011 at 21:21
  • $\begingroup$ You probably want to say that the thermodynamic entropy tells you how much of the energy is not available for work! I'll also go with Matt's answer than this only makes sense if the state is in thermal equilibrium, whereas Shannon entropy is more general. $\endgroup$
    – Earl
    Commented Oct 5, 2011 at 21:25
  • $\begingroup$ @Aaron: Actually, the reconciliation is that I was answering yes to whether or not the two entropy definitions agree. I didn't mean that entropy measures extractable work. $\endgroup$ Commented Oct 6, 2011 at 4:08

3 Answers 3


UPDATE: Below I am answering yes to the first question in the post (are the two kinds of entropy the same up to a constant). This has led to some confusion as both Matt and John gave answers saying "the answer is no", however I believe they are referring to the title "Does entropy measure extractable work?". Although the author uses the two interchangeably, the question itself contains a false premise: namely that physical entropy is a measure of extractable work ("Entropy, as seen from thermodynamics, measures the amount of extractable work"). This is simply not true in the general case, though can be true for certain special cases, as Matt's counter example shows. A concrete case of this is a ball placed anywhere on a level surface. If the ball is placed randomly, no more work can be extracted than if its location is known.

The answer is yes, the two kinds of entropy are the same up to a constant factor of $k_B \log 2$ (which is also the origin of Landauer's principle). On way to see this is via Maxwell's demon. In particular, via the Szilard engine, an idealised heat engine that uses a single particle gas. You then introduce a partition, which effectively partitions the gas into two regions, only one of which contains the particle. Now, if you knew which side of the partition the particle was on, you could use the pressure difference to extract work, and if you don't you can't since you don't know which way it will push.

Now the connection with information theory comes in when we measure which side of the partition the particle is on. From this we are gaining a certain amount of entropy (and hence information) in the register which holds our measurement result. But having this information decreases the entropy of the gas. And hence you can go back and forth between information entropy and physical entropy.

There is a fairly extensive literature on the matter, so instead of trying to give you a list, I'll point you to a review article on Maxwell's Demon and information theory from a few years ago: arXiv:0707.3400,

  • $\begingroup$ Yes, I see. But still with the Szilard engine example, I'd like to go further. If I know in which side the particle si, I can extract kT ln(2) of work. If I don't know where the particle is, I can get no work. But, what if I have partial knowledge? Is still Shannon's entropy the answer? $\endgroup$ Commented Sep 16, 2011 at 14:14
  • 1
    $\begingroup$ @Javier: Yes, you can use exactly the same trick, but use a measurement that returns 0 if the particle is in the left most compartment, and 1 with probability $p$ if the particle is in the rightmost compartment and 0 otherwise, and proceed from there. Have a look at the paper, there's lots of good stuff in it. $\endgroup$ Commented Sep 16, 2011 at 14:30
  • $\begingroup$ More recent results here arxiv.org/abs/0908.0424 and here arxiv.org/abs/1009.1630 $\endgroup$
    – Marco
    Commented Sep 29, 2011 at 7:20

The answer is no. Consider a system that has a degenerate ground state, such that the density matrix is a mixture of two ground state eigenstates. This has nonzero Shannon entropy, but you can't extract any work from it. More generally, thermodynamic entropy is not really well-defined for nonequilibrium systems, but Shannon entropy is.

My own take is that thermodynamic entropy and Shannon entropy are two conceptually distinct things. They happen to coincide in a wide variety of circumstances, but not always. It is not even clear to me whether the cases in which they happen to coincide in classical and quantum theory are necessary coincidences. It may be possible to come up with a well-defined physical theory in which they never coincide, e.g. in the convex set famework for generalized probabilistic theories studied in the quantum foundations community.

  • $\begingroup$ A neat counterexample Matt! I was thinking a similar thing when reading the question. $\endgroup$
    – Earl
    Commented Oct 5, 2011 at 21:20
  • $\begingroup$ @Matt: Good answer, and you get +1 from me, but I do not understand the claim that thermodynamic entropy is not well defined for non-equilibrium systems. It's given by $-k_B \sum_i p_i\log p_i$ where $p_i$ is the probability of the $i^{th}$ microstate. This is obviously defined for non-equilibrium systems and equilibrium systems alike. $\endgroup$ Commented Oct 6, 2011 at 4:27
  • $\begingroup$ Actually I guess the issue is clouded by the existence of alternative non-equivalent definitions of entropy. $\endgroup$ Commented Oct 6, 2011 at 5:55
  • $\begingroup$ @JoeFitzsimons You've likely worked this out since Oct 2 2011, but in case you haven't, the "experimental" entropy (that computed from macrostate properties) becomes fuzzy because things like $T$ become fuzzy. It is equal to your expression when $i$ stands for state of constituent particle as well as global microstate (if, for the former, you multiply by the number of particles), i.e. constituent particle microstates are statistically independent. Your expression is in general equal to the Shannon entropy - when there are correlations the probabilities of microstates become .... $\endgroup$ Commented Nov 1, 2013 at 23:47
  • $\begingroup$ @JoeFitzsimons ... impossible to access through macroscopic measurements. If you've not seen this paper before, you might find it helpful. E. T. Jaynes, "Gibbs vs Boltzmann Entropies", Am. J. Phys. 33, number 5, pp 391-398, 1965 as well as many other of his works in this field. BTW Yours is an excellent question. $\endgroup$ Commented Nov 1, 2013 at 23:50

Because the word "work" has multiple meanings, the answer in general is "no".

Typically the first meaning taught to students is thermodynamic, but (as Dirac demonstrated) a generalized meaning (which includes thermodynamic work as a particular case) amounts to:

Given a class of dynamical processes, 'work' is any potential function that naturally describes what that class accomplishes.

Specifically, in the context of isotope separation, Dirac established that the work potential $\mathcal{V}_c$ that is naturally associated to an isotope concentration $c$ is given by

$\displaystyle\qquad \mathcal{V}_c(c) = (2 c' - 1) \log\left[\frac{c'}{1-c'}\right]$

or equivalently for spin polarization $p = 2 c - 1$ the Dirac value function $\mathcal{V}_p$ associated to separative transport of spin polarization is

$\displaystyle\qquad\mathcal{V}_p(p') = \mathcal{V}_c(c')\big|_{c' = (1+p')/2} = p' \log\left[\frac{1+p'}{1-p'}\right]$

The key point is Dirac's value function is not proportional to an entropy difference (as is evident because $0\le \mathcal{V}_p(p') \lt \infty$ while per-mole entropy ranges over a finite range).

Moreover, the Dirac work associated to separation cannot be reversed to return mechanical work, since the separation process is entropically irreversible. Nonetheless, Dirac work has substantial economic value, and in fact defines the unit of value of a global market in separative work units (SWUs, pronounced "swooz").

For a derivation of the Dirac work function, see Dirac's own (unpublished) technical note "Theory of the separation of isotopes by statistical methods" (circa 1940), which appears in his Collected Works, or alternatively Donald Olander's survey article "Technical Basis of the Gas Centrifuge" (1972), or in general any textbook on isotope separation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.