Skip to main content
unfortunate typo
Source Link
N. Virgo
  • 34.9k
  • 7
  • 105
  • 159

You have to be careful when thinking about this. For example, you talk about "the entropy of a message", but what could that mean? Shannon's entropy is a property of a probability distribution, but a message isn't a probability distribution, so a message does not in itself have an entropy.

The entropy only comes in when you don't know which message will be sent. For example: suppose you ask me a question to which the possible answers are "yes" and "no", and you have no idea what my answer will be. Because you don't know the answer, you can use a probability distribution: $p(\text{yes})=p(\text{no})=1/2,$ which has an entropy of one bit. Thus when I give my answer, you receive one bit of entropyinformation. On the other hand, if you ask me a question to which you already know the answer, my reply gives you no information. You can see this by noting that the probability distribution $p(\text{yes})=1; \,\,p(\text{no})=0$ has an entropy of zero.

Now, in these examples the entropy is equal to the information gained - but in a sense they are equal and opposite. Before you receive the message there is entropy, but afterwords there is none. (If you ask the same question twice you will not receive any more information.) The entropy represents your uncertainty, or lack of information about the message, before you receive it, and this is precisely why it is equal to the amount of information that you gain when you do receive the message.

In physics it is the same. The physical entropy represents a lack of information about a system's microscopic state. It is equal to the amount of information you would gain if you were to suddenly become aware of the precise position and velocity of every particle in the system* --- but in physics there is no way that can happen. Measuring a system can give us at most a few billions of bits (usually far fewer), but the entropy of a macroscopically sized system is a lot larger than this, of the order $10^{23}$ bits or more.

The second law of thermodynamics arises because there are a lot of ways we can lose information about a system, for example if the motions of its particles become correlated with the motions of particles in its surroundings. This increases our uncertainty about the system, i.e. its entropy. But the only way its entropy can decrease is if we make a measurement, and this decrease in entropy is typically so small it can be neglected.

If you would like to have a deep understanding of the relationship between Shannon entropy and thermodynamics, it is highly recommended that you read this long but awesome paper by Edwin Jaynes.

* or, if we're thinking in terms of quantum mechanics rather than classical mechanics, it's the amount of information you would gain if you made a measurement such that the system was put into a pure state after the measurement.

You have to be careful when thinking about this. For example, you talk about "the entropy of a message", but what could that mean? Shannon's entropy is a property of a probability distribution, but a message isn't a probability distribution, so a message does not in itself have an entropy.

The entropy only comes in when you don't know which message will be sent. For example: suppose you ask me a question to which the possible answers are "yes" and "no", and you have no idea what my answer will be. Because you don't know the answer, you can use a probability distribution: $p(\text{yes})=p(\text{no})=1/2,$ which has an entropy of one bit. Thus when I give my answer, you receive one bit of entropy. On the other hand, if you ask me a question to which you already know the answer, my reply gives you no information. You can see this by noting that the probability distribution $p(\text{yes})=1; \,\,p(\text{no})=0$ has an entropy of zero.

Now, in these examples the entropy is equal to the information gained - but in a sense they are equal and opposite. Before you receive the message there is entropy, but afterwords there is none. (If you ask the same question twice you will not receive any more information.) The entropy represents your uncertainty, or lack of information about the message, before you receive it, and this is precisely why it is equal to the amount of information that you gain when you do receive the message.

In physics it is the same. The physical entropy represents a lack of information about a system's microscopic state. It is equal to the amount of information you would gain if you were to suddenly become aware of the precise position and velocity of every particle in the system* --- but in physics there is no way that can happen. Measuring a system can give us at most a few billions of bits (usually far fewer), but the entropy of a macroscopically sized system is a lot larger than this, of the order $10^{23}$ bits or more.

The second law of thermodynamics arises because there are a lot of ways we can lose information about a system, for example if the motions of its particles become correlated with the motions of particles in its surroundings. This increases our uncertainty about the system, i.e. its entropy. But the only way its entropy can decrease is if we make a measurement, and this decrease in entropy is typically so small it can be neglected.

If you would like to have a deep understanding of the relationship between Shannon entropy and thermodynamics, it is highly recommended that you read this long but awesome paper by Edwin Jaynes.

* or, if we're thinking in terms of quantum mechanics rather than classical mechanics, it's the amount of information you would gain if you made a measurement such that the system was put into a pure state after the measurement.

You have to be careful when thinking about this. For example, you talk about "the entropy of a message", but what could that mean? Shannon's entropy is a property of a probability distribution, but a message isn't a probability distribution, so a message does not in itself have an entropy.

The entropy only comes in when you don't know which message will be sent. For example: suppose you ask me a question to which the possible answers are "yes" and "no", and you have no idea what my answer will be. Because you don't know the answer, you can use a probability distribution: $p(\text{yes})=p(\text{no})=1/2,$ which has an entropy of one bit. Thus when I give my answer, you receive one bit of information. On the other hand, if you ask me a question to which you already know the answer, my reply gives you no information. You can see this by noting that the probability distribution $p(\text{yes})=1; \,\,p(\text{no})=0$ has an entropy of zero.

Now, in these examples the entropy is equal to the information gained - but in a sense they are equal and opposite. Before you receive the message there is entropy, but afterwords there is none. (If you ask the same question twice you will not receive any more information.) The entropy represents your uncertainty, or lack of information about the message, before you receive it, and this is precisely why it is equal to the amount of information that you gain when you do receive the message.

In physics it is the same. The physical entropy represents a lack of information about a system's microscopic state. It is equal to the amount of information you would gain if you were to suddenly become aware of the precise position and velocity of every particle in the system* --- but in physics there is no way that can happen. Measuring a system can give us at most a few billions of bits (usually far fewer), but the entropy of a macroscopically sized system is a lot larger than this, of the order $10^{23}$ bits or more.

The second law of thermodynamics arises because there are a lot of ways we can lose information about a system, for example if the motions of its particles become correlated with the motions of particles in its surroundings. This increases our uncertainty about the system, i.e. its entropy. But the only way its entropy can decrease is if we make a measurement, and this decrease in entropy is typically so small it can be neglected.

If you would like to have a deep understanding of the relationship between Shannon entropy and thermodynamics, it is highly recommended that you read this long but awesome paper by Edwin Jaynes.

* or, if we're thinking in terms of quantum mechanics rather than classical mechanics, it's the amount of information you would gain if you made a measurement such that the system was put into a pure state after the measurement.

Source Link
N. Virgo
  • 34.9k
  • 7
  • 105
  • 159

You have to be careful when thinking about this. For example, you talk about "the entropy of a message", but what could that mean? Shannon's entropy is a property of a probability distribution, but a message isn't a probability distribution, so a message does not in itself have an entropy.

The entropy only comes in when you don't know which message will be sent. For example: suppose you ask me a question to which the possible answers are "yes" and "no", and you have no idea what my answer will be. Because you don't know the answer, you can use a probability distribution: $p(\text{yes})=p(\text{no})=1/2,$ which has an entropy of one bit. Thus when I give my answer, you receive one bit of entropy. On the other hand, if you ask me a question to which you already know the answer, my reply gives you no information. You can see this by noting that the probability distribution $p(\text{yes})=1; \,\,p(\text{no})=0$ has an entropy of zero.

Now, in these examples the entropy is equal to the information gained - but in a sense they are equal and opposite. Before you receive the message there is entropy, but afterwords there is none. (If you ask the same question twice you will not receive any more information.) The entropy represents your uncertainty, or lack of information about the message, before you receive it, and this is precisely why it is equal to the amount of information that you gain when you do receive the message.

In physics it is the same. The physical entropy represents a lack of information about a system's microscopic state. It is equal to the amount of information you would gain if you were to suddenly become aware of the precise position and velocity of every particle in the system* --- but in physics there is no way that can happen. Measuring a system can give us at most a few billions of bits (usually far fewer), but the entropy of a macroscopically sized system is a lot larger than this, of the order $10^{23}$ bits or more.

The second law of thermodynamics arises because there are a lot of ways we can lose information about a system, for example if the motions of its particles become correlated with the motions of particles in its surroundings. This increases our uncertainty about the system, i.e. its entropy. But the only way its entropy can decrease is if we make a measurement, and this decrease in entropy is typically so small it can be neglected.

If you would like to have a deep understanding of the relationship between Shannon entropy and thermodynamics, it is highly recommended that you read this long but awesome paper by Edwin Jaynes.

* or, if we're thinking in terms of quantum mechanics rather than classical mechanics, it's the amount of information you would gain if you made a measurement such that the system was put into a pure state after the measurement.