Entropy and Information

Question

Several posts and my classes in thermodynamics equate increase in entropy with loss of information. Shannon clearly showed that the information content of a message is zero when its entropy is zero and that its information content increases with increasing entropy. So entropy increase leads to more information, which is consistent with the evolution of the universe from a disordered plasma to one that contains lots of order. Why does physics continue to get the relationship between entropy and information backwards?

Answer 1

You have to be careful when thinking about this. For example, you talk about "the entropy of a message", but what could that mean? Shannon's entropy is a property of a probability distribution, but a message isn't a probability distribution, so a message does not in itself have an entropy.

The entropy only comes in when you don't know which message will be sent. For example: suppose you ask me a question to which the possible answers are "yes" and "no", and you have no idea what my answer will be. Because you don't know the answer, you can use a probability distribution: $p(\text{yes})=p(\text{no})=1/2,$ which has an entropy of one bit. Thus when I give my answer, you receive one bit of information. On the other hand, if you ask me a question to which you already know the answer, my reply gives you no information. You can see this by noting that the probability distribution $p(\text{yes})=1; \,\,p(\text{no})=0$ has an entropy of zero.

Now, in these examples the entropy is equal to the information gained - but in a sense they are equal and opposite. Before you receive the message there is entropy, but afterwords there is none. (If you ask the same question twice you will not receive any more information.) The entropy represents your uncertainty, or lack of information about the message, before you receive it, and this is precisely why it is equal to the amount of information that you gain when you do receive the message.

In physics it is the same. The physical entropy represents a lack of information about a system's microscopic state. It is equal to the amount of information you would gain if you were to suddenly become aware of the precise position and velocity of every particle in the system* --- but in physics there is no way that can happen. Measuring a system can give us at most a few billions of bits (usually far fewer), but the entropy of a macroscopically sized system is a lot larger than this, of the order $10^{23}$ bits or more.

The second law of thermodynamics arises because there are a lot of ways we can lose information about a system, for example if the motions of its particles become correlated with the motions of particles in its surroundings. This increases our uncertainty about the system, i.e. its entropy. But the only way its entropy can decrease is if we make a measurement, and this decrease in entropy is typically so small it can be neglected.

If you would like to have a deep understanding of the relationship between Shannon entropy and thermodynamics, it is highly recommended that you read this long but awesome paper by Edwin Jaynes.

^{* or, if we're thinking in terms of quantum mechanics rather than classical mechanics, it's the amount of information you would gain if you made a measurement such that the system was put into a pure state after the measurement.}

Answer 2

I believe conflicting concepts of information and entropy under the same names are at work here.

In information theory, the information content of a message is naturally the amount of information you get from decoding the message. Entropy, as defined in information theory, is a measure of how random the message is, which is precisely the information content of the message, as the more random a message is, the more information will be gained from decoding the message.

In physics, I assume that information refers to information about the exact microscopic configuration (microstate) of the system. Entropy is then a measure of how probable the macroscopic state of the system is. It happens that systems with high entropy turn out to be more 'random' states too, usually states of equilibrium. Naturally for these states, more microscopic configurations with all macroscopic variables and the macroscopic state unchanged are possible than other macroscopic states (precisely why the macroscopic state is more probable). This would thus correspond to little information known regarding which exact microscopic configuration the system is, given the macroscopic state of the system with its known macroscopic variables. Correspondingly, low entropy systems would have more information known regarding the microscopic state of the system.

Thus I believe these are two different concepts, and should not be confused.

However, these concepts are certainly deeply related (they do go by same names). To reconcile your apparent paradox, assume we have a high entropy physical system with known macroscopic variables. We treat this system as a information theoretical message. In accordance to physics, we have little information about which microscopic configuration the system is in. Now, we 'decode' the 'message' aka we somehow find out which specific microscopic configuration the system is in. For high entropy systems, the information gained would be high, and similarly it would be low for low entropy systems, in accordance to information theory. Paradox resolved!

Answer 3

It isn't a backwards definition in physics. It is more of a paradox in information theory.

If you have a message (some sequence of bits), it has some measure of entropy. If you then apply a (lossless) compression scheme to it, you are keeping the amount of information constant, but reducing the number of bits used to represent it. If you use the most efficient compression scheme possible, the result will be a stream of bits which will have the appearance of randomness in that inspection of each successive bit of the compressed message does not allow you to predict the value of the next bit in the message. The message will have maximal information content in that the fewest possible bits are required to represent the message, but the message itself becomes indistinguishable from a random stream of bits (which would have maximum entropy).

Answer 4

Just wanted to point out that the last comment may mislead people. "Entropy, as defined in information theory, is a measure of how random the message is..." Perhaps the writer means a measure of how unpredictable it is, and uses random as a synonym. It's not a good one. If a signal is predictable it is redundant, and redundancy, under scarce resources, is waste. A sentence "dog bites man" has more entropy than "man bites dog," because the latter is more unexpected. Information is found in unexpected signals, which is why newspapers will print an article about a man biting a dog, not the other way around. So if decoding cryptographic signals, the apparently random inputs are not entropy, but information. Once the signals are completely decoded, putting aside any semantic content, they are entropy to the code breakers.

Shannon's entropy is a byproduct of information, which, like happiness, is pursued. The act of figuring out what something means, the "aha" or "viola" moment, is information. But once figured out, its not information. What is a purely random stream of signals? Entropy, of course, because it offers no information. Information is best served hot: a really difficult code will provide little, until suddenly a major break is made.

Confusion about randomness arises when we mix signals and semantics. The code-breaker sees decoded signals as entropy, but the semantic content in those signals may be very unexpected to the people who read it, real information.

Macroscopic entropy, generated by energy use, increases with time. It is 'relativistic' in the sense that it is defined in a local reference frame. But since quantum dynamics and general relativity remain unresolved, we can't say it's impossible that decoherence generates enough information to change the big picture.