Skip to main content
Physics

Return to Answer

Mod Moved Comments To Chat
added 130 characters in body
Source Link
knzhou
knzhou
  • 105.1k
  • 24
  • 297
  • 494

Formally, the two entropies are the same thing. The Gibbs entropy, in thermodynamics, is $$S = -k_B \sum p_i \ln p_i$$ while the Shannon entropy of information theory is $$H = -\sum p_i \log_2 p_i.$$ These are equal up to some numerical factors. Given a statistical ensemble, you can calculate its (thermodynamic) entropy using the Shannon entropy, then multiplying by constants.

However, there is a sense in which you're right. Often when people talk about Shannon entropy, they only use it to count things that we intuitively perceive as information. For example, one might say the entropy of a transistor, flipped to 'on' or 'off' with equal likelihood, is 1 bit.

But the thermodynamic entropy of the transistor is thousands, if not millions of times higher, because it counts everything, i.e. the configurations of all the atoms making up the transistor. (If you want to explain it to your programmer colleagues, say they're not counting whether each individual atom is "on" or "off".)

In general, the amount of "intuitive" information (like bits, or words in a book) is a totally negligible fraction of the total entropy. The thermodynamic entropy of a library is about the same as that of a warehouse of blank books.

Formally, the two entropies are the same thing. The Gibbs entropy, in thermodynamics, is $$S = -k_B \sum p_i \ln p_i$$ while the Shannon entropy of information theory is $$H = -\sum p_i \log_2 p_i.$$ These are equal up to some numerical factors. Given a statistical ensemble, you can calculate its (thermodynamic) entropy using the Shannon entropy, then multiplying by constants.

However, there is a sense in which you're right. Often when people talk about Shannon entropy, they only use it to count things that we intuitively perceive as information. For example, one might say the entropy of a transistor, flipped to 'on' or 'off' with equal likelihood, is 1 bit.

But the thermodynamic entropy of the transistor is thousands, if not millions of times higher, because it counts everything, i.e. the configurations of all the atoms making up the transistor.

In general, the amount of "intuitive" information (like bits, or words in a book) is a totally negligible fraction of the total entropy. The thermodynamic entropy of a library is about the same as that of a warehouse of blank books.

Formally, the two entropies are the same thing. The Gibbs entropy, in thermodynamics, is $$S = -k_B \sum p_i \ln p_i$$ while the Shannon entropy of information theory is $$H = -\sum p_i \log_2 p_i.$$ These are equal up to some numerical factors. Given a statistical ensemble, you can calculate its (thermodynamic) entropy using the Shannon entropy, then multiplying by constants.

However, there is a sense in which you're right. Often when people talk about Shannon entropy, they only use it to count things that we intuitively perceive as information. For example, one might say the entropy of a transistor, flipped to 'on' or 'off' with equal likelihood, is 1 bit.

But the thermodynamic entropy of the transistor is thousands, if not millions of times higher, because it counts everything, i.e. the configurations of all the atoms making up the transistor. (If you want to explain it to your programmer colleagues, say they're not counting whether each individual atom is "on" or "off".)

In general, the amount of "intuitive" information (like bits, or words in a book) is a totally negligible fraction of the total entropy. The thermodynamic entropy of a library is about the same as that of a warehouse of blank books.

Source Link
knzhou
knzhou
  • 105.1k
  • 24
  • 297
  • 494

Formally, the two entropies are the same thing. The Gibbs entropy, in thermodynamics, is $$S = -k_B \sum p_i \ln p_i$$ while the Shannon entropy of information theory is $$H = -\sum p_i \log_2 p_i.$$ These are equal up to some numerical factors. Given a statistical ensemble, you can calculate its (thermodynamic) entropy using the Shannon entropy, then multiplying by constants.

However, there is a sense in which you're right. Often when people talk about Shannon entropy, they only use it to count things that we intuitively perceive as information. For example, one might say the entropy of a transistor, flipped to 'on' or 'off' with equal likelihood, is 1 bit.

But the thermodynamic entropy of the transistor is thousands, if not millions of times higher, because it counts everything, i.e. the configurations of all the atoms making up the transistor.

In general, the amount of "intuitive" information (like bits, or words in a book) is a totally negligible fraction of the total entropy. The thermodynamic entropy of a library is about the same as that of a warehouse of blank books.