How is thermodynamic entropy defined? What is its relationship to information entropy?

Question

I read that thermodynamic entropy is a measure of the number of microenergy states. What is the derivation for $S=k\log N$, where $k$ is Boltzmann constant, $N$ number of microenergy states.

How is the logarithmic measure justified?

Does thermodynamic entropy have anything to do with information entropy (defined by Shannon) used in information theory?

Answer 1

I think that the best way to justify the logarithm is that you want entropy to be an extensive quantity -- that is, if you have two non-interacting systems A and B, you want the entropy of the combined system to be $$ S_{AB}=S_A+S_B. $$ If the two systems have $N_A,N_B$ states each, then the combined system has $N_AN_B$ states. So to get additivity in the entropy, you need to take the log.

You might wonder why it's so important that the entropy be extensive (i.e., additive). That's partly just history. Before people had worked out the microscopic basis for entropy, they'd worked out a lot of the theory on macroscopic thermodynamic grounds alone, and the quantity that they'd defined as entropy was additive.

Also, the number of states available to a macroscopic system tends to be absurdly, exponentially large, so if you don't take logarithms it's very inconvenient: who wants to be constantly dealing with numbers like $10^{10^{20}}$?

Answer 2

The thermodynamic entropy is not what you wrote. That is the information entropy. Microstates and the counting of are not thermodynamic concepts, but rather statistical mechanics ones. Thermodynamics proceeds by identifying reversible processes, and goes on to study them. A key point is noticing that there are quantities called state variables which are intrinsic to a substance, which do not depend on the path used to obtain it, such as temperature, or pressure. Heat transfer is explicitly (and experimentally) not one such quantity. However, by considering a perfect reversible heat engine, it is possible to show that the quantity $dQ/T$ is a state variable. Integrating it gives $$\Delta S = \int \frac{dQ}{T}$$ and is the change of entropy. Thermodynamically one can only talk of changes, and the zero is not particularly well defined --- the 3rd law of thermodynamics tries to define it, but it is not always applicable.

Answer 3

The connection between thermodynamic and information theoretic entropies is a deep connection, and has been an active research area for more than a century.

To get more information, you can look at

• This well written comprehensive review at Stanford Encyclopedia of Philosophy, including the link with macroscopic thermodynamical entropy.
• This webpage and this wikipedia page on the subject give shorter overviews of some of the links between these entropies.

Various Researches on Maxwell's demon,Szilard's engine and Landauer's principle can also be helpful.

Answer 4

The reason for the logarithm in the statistical definition of entropy is that it makes the entropy additive and the number of microstates multiplicative. This means that if you have subsystems A & B, total number of microstates is the product of the subsystem’s number of microstates, and the entropy is additive over subsystems due to the properties of the logarithm. A way to see the relation of thermodynamic entropy and statistical entropy is to picture the volume occupied by a system of classical particles as partitioned in cells of very small volume. Then you start counting the number of ways to distribute N particles in the cells. The number of microstates goes like the numbers of cells to the power of the number of particles. The number of cells is the volume divided by the number of partitions. Taking the logarithm of this expression you get number of particles multiplied by the logarithm of the volume over the number of partitions. Multiplying by k you get an expression for the entropy. Then the change of entropy, for a process where volume changes, is obtained by subtracting the final entropy from the initial entropy. The number of partitions is cancelled from the equation by the properties of logarithms. So this is another reason for the logarithm in the expression. What you get finally is the Boltzmann constant multiplied by the number of particles and the logarithm of final volume over initial volume. The way this result is obtained makes clear that the change in entropy is path independent. The same expression for the change in entropy can be obtained, using thermodynamics, for the reversible isothermal expansion of an ideal gas, by dividing the heat absorbed over the temperature. If a process is not isothermal then you can break it in small nearly isothermal infinitesimal paths, then integrate the change in entropy. This is the easiest way to answer your question. Similar arguments can be used for quantum systems but they are more complex.

Answer 5

Hey you have the Thermodynamic definition which is :

$$\Delta S = \int \frac{dQ}{T}$$

and the statistical definition:

$S=k\log N$

N- the number of the possibly states of the system (including degeneration).

For mostly everything they are equivalent!!

note that you can not measure enytopy, but you can measure the $\Delta$ (the change) in the entropy of the system.