Confusion regarding entropy, reference papers request

Question

1.a In paragraph at Wikipedia/Entropy it is stated:

This is because energy supplied at a high temperature (i.e. with low entropy) tends to be more useful than the same amount of energy available at room temperature

1.b But at What is Entropy? there is:

The higher the temperature of the gas, the faster the gas particles are moving on average, so the wider the range of possible velocities for the gas particles, and hence, the more uncertainty we have about the velocity of any particular particle. Thus, higher temperature, as well as greater volume, mean higher entropy.

2.a I was trying to obtain information on relation between entropy and work. Basically, if an external process modifies a system so that its entropy is reduced, then it is expected by me that work was done on the system and energy was contributed to it. I received good answers ( Connection between entropy and energy ). However I feel that I was not properly understood.

2.b In 1953 paper of L. Brillouin ("The Negentropy Principle of Information") wrote what I basically had in mind when I asked the question. I am not aware if the approach is correct and accepted.

An isolated system contains negentropy if it reveals a possibility for doing mechanical or electrical work. If the system is not at a uniform temperature T, but consists of different parts at different temperatures, it contains a certain amount of negentropy. This negentropy can be used to obtain some mechanical work done by the system, or it can be simply dissipated and lost by thermal conduction. A difference in pressure between different parts of the system is another case of negentropy. A difference of electrical potential repre-sents another example. A tank of compressed gas in a room at atmospheric pressure, a vacuum tank in a similar room, a charged battery, any device that can produce high-grade energy (mechanical work) or be degraded by some irreversible process (thermal con-duction, electrical resistivity, friction, viscosity) is a source of negentropy.

3.a There is often a contradict intuition about information entropy. Shannon approach sees maximum entropy as maximum uncertainty / information.

3.b Joe Rosen in "The Symmetry Principle" states that maximum entropy is maximum symmetry – which means maximum redundancy, so minimum amount of information.

4. Non-equilibrium thermodynamics – the branch is not mature and does not serve as a tool to solve issues like in 2.a, AFAIK.

I was seeing The Physics of Maxwell's demon and information as very good guide through equilibrium (what about non-equilibrium?) thermodynamics, its connection with work and information. The authors did, however, when elaborating on entropy of a system in Brillouin's view, put an equal sign to it and Shannon entropy – which I think is contradict to Brillouin itself – who as I cited in 2.b stated that it is disequilibrium that allows system to work – meaning a complex, information-rich system (in other paragraph he also explicitly sates that negentropy is information). This conforms to Rosen 3.b.

Are there other good compilations on entropy, information, maybe complexity science? Such that solve the contradictions, form compatible non-equilibrium foundation?

Answer 1

For clear expositions on the relationship between entropy and information, and the foundations of the non-equilibrium theory, it's well worth surfing through papers by Edwin Jaynes. You can find his full bibliography here. It's probably best to start with one of his later papers, since they take a more pedagogical approach. I would recommend 'The Evolution of Carnot's Principle' as a starting point, and then maybe 'The Gibbs Paradox'.

Now, to answer some of your more specific points.

1. Entropy can be defined, at least to start with, by saying that when a body at a temperature $T$ gains an amount $Q$ of heat, its entropy increases by $Q/T$. So for a given amount of heat, a higher temperature means a lower entropy. However, it doesn't necessarily follow that bodies (or gases) with a higher temperature always have lower entropy than bodies at a lower temperature, because hotter bodies also have more energy. Both quotes are correct, because the first is talking about entropy per unit energy, and the second is talking about entropy per unit volume.

The really important thing, though, isn't how much entropy a body has at a given temperature, but how much its entropy changes if it gains or loses some energy. If you have an isolated system composed of a hot body (at temperature $T_H$) and a cold one (at $T_C$), some heat $Q$ will flow from the hot body to the cold one. The entropy of the hot body changes by $-Q/T_H$ (i.e. its entropy decreases), but the entropy of the cold body increases by $Q/T_C$, which is bigger because $T_H>T_C$.

The total entropy change in this system is $\frac{Q}{T_C}-\frac{Q}{T_H}$. If the two temperatures are equal then the entropy is maximised and no further heat transfer can happen.

2.a A nice way to think of work is to think of it as energy without any associated entropy. It's like heat with an infinite temperature, so that $Q/T$ goes to 0. Let's say we have a body at some temperature $T$ and we want to take some heat out of it and convert it to some work $W$. The entropy change of this transformation is $$\text{entropy of the work}-\text{entropy of the heat}=0-W/T<0,$$ which means that it's impossible to perform this transformation unless in doing so we increase the entropy of some other system by the same amount ($W/T$) or more, so that the total entropy change is greater than zero.

In particular, if we have two bodies at different temperatures, we can increase the entropy of the cold one by transferring some heat into it from the hot one. If we put an amount $Q$ into the cold body we must take an amount $Q+W$ from the hot one. (That's the energy we're putting into the cold body, plus the amount we're converting into work.) The entropy of the cold body now increases by $Q/T_C$ while the hot body's entropy decreases by $(Q+W)/T_H$. The total entropy change is therefore $$ \frac{Q}{T_C}-\frac{Q+W}{T_H}, $$ which is positive as long as $W<Q\left(1-\frac{T_C}{T_H}\right)$, which is called the Carnot limit.

To reiterate, the point is that work is energy with no associated entropy, and if you want to create if from heat, you have to do it by increasing the entropy of some other system in order to satisfy the second law.

But please note: your intuition about the relationship between work and entropy was not quite correct. Doing work on a system can decrease its entropy (though it can also increase it, by adding energy), but doing work on a system is not the only way to decrease its entropy. You can also do this by removing heat, for example, as I showed above.

2.b This quote is correct. If a system has 'negentropy' (which isn't a very commonly used term, but it's a perfectly reasonable one) then its entropy isn't at the maximum, which means it's possible to increase its entropy in order to convert some heat into work.

3.a Be careful here - you seem to be equating uncertainty with information, but they're actually opposites! The less information you have about something, the more uncertain you are about it. Entropy is uncertainty - specifically, it's the amount of uncertainty you have about a system's microscopic state if you know its macroscopic properties (temperature, pressure, volume, etc.). It can be expressed in the same units as information (e.g. bits) but it has the opposite sign, so an increase in entropy is a loss of information.

3.b I haven't read this source, so I can't really comment on the symmetry part of it, but the idea that maximum entropy equals minimum information is correct.

4. Non-equilibrium thermodynamics is pretty mature these days. The basic idea is that you take the stuff I was talking about above, and instead of a finite amount of heat $Q$, you consider a flow of heat $dQ/dt$. This leads to a continuous change in entropy $dS/dt$, which must be positive for an isolated system. The maths is not very different from equilibrium thermodynamics. If you have some specific questions about it I can probably help to answer them.

Answer 2

Entropy is a broad topic, and very important both to physics and information theory.

In general entropy is a measure of not knowing things. So a state of high entropy is when there are many possible sequences or many possible micro states of a physical system.

In information theory it is $$S(\{p_i\}_i) = - \sum_i p_i \log(p_i).$$ In particular, when it is high it mean that measuring exact $i$ gives a lot of information.

In thermodynamics, it is related to the volume of the phase-space. In fact it is the same formula as in information theory, but you measure probability distribution for particles being in state $(x_1,\ldots,x_n,p_1,\ldots,p_n)$, where $x_i$ are positions and $p_i$ - momenta.

When it comes to entropy-energy, look at thermodynamics. For open system entropy may decrease with temperature as particles are flying away.

Possible readings (as the questions you asked cannot be answered in one post):

• Kerson Huang, Introduction to Statistical Physics
• T.M. Cover, J.A. Thomas, Elements of information theory
• A. Wehrl, General properties of entropy, Reviews of Modern Physics, 1978

Answer 3

Re items 1, the quoted statement in item 1a is at worst incorrect, and at best incomplete (if one infers from the term "high temperature" that a lower temperature heat sink is also present). The "high temperature" energy is only useful if such a lower temperature heat sink is present, and the total entropy of those two reservoirs is less than if they were allowed to equilibrate at the same temperature. For example, for two reservoirs of ideal gas with equal volumes $V$ and numbers of molecules $N$, one at temperature $T+\Delta T$ and the other at $T-\Delta T$, the total entropy of the two is (ref Feynman lectures vol. 1, lecture 44): \begin{align} S &= 2a + Nk \left[ 2\ln(V) + \frac{1}{\gamma -1} \left[ \ln (T+\Delta T) +\ln (T-\Delta T) \right] \right]\\&= 2a + Nk \left[ 2\ln(V) + \frac{1}{\gamma -1} \ln (T^2-\Delta T^2) \right]\end{align} so the entropy is maximized when $\Delta T=0$ (i.e. when the two reservoirs are at the same temperature).

The statement in item 1b) is correct: one can increase an item's temperature by adding heat to it, thereby increasing its entropy.

Re items 2, in the context of the same two ideal gas reservoirs, now at the same temperature, your idea of doing work to reduce their entropy could be realized by connecting a reversible heat engine between the two, and driving that engine with work to transfer heat from one reservoir to the other, cooling one and heating the other (i.e. a heat pump, or refrigerator). I'm unfamiliar with negentropy, but the Brillouin quote appears to be generalizing the above discussion: a difference between two subsections of the system is necessary to extract work.

Re items 3 & 4, Shannon's equation for information entropy (see Piotr Migdal's answer) uses essentially the same form as in statistical mechanics (which is why Shannon borrowed the name). I like the quote from Pierce's "An Introduction to Information Theory": "Once we understand entropy as it is used in communication theory thoroughly, there is no harm in trying to relate it to the entropy in physics, but the literature indicates that some workers have never recovered from the confusion engendered by an early admixture of ideas concerning the entropies of physics and communication theory." Learn one, then the other...

I would recommend the Feynman lecture referenced above as well as the subsequent lectures on thermo in the same volume.