How do physicists mathematically define entropy (for the second law of thermodynamics) and how is it related to statistical definitions of entropy?

Even though there are many questions on this site about entropy (such as this one), none I could find was mathematically rigorous or had a complete rigorous answer.

I am looking for precise answers that can be understood by mathematicians.


In mathematical statistics, we have many different definitions of the entropy of (or between) probability distributions. Notable ones are:

  • The $\alpha$-entropy $N(\alpha)$ of a distribution $\rho$ on the integers, which is defined as $$\log \sum_{n \in \mathbb{N}} \rho(n)^\alpha.$$ It can be extended to the entropy of a distribution defined on any separable metric space.
  • The Kullback-Leibler divergence (or relative entropy) $$D_{KL}(P, Q) = \int \log\frac{dP}{dQ} dP.$$

Note that transforms $T$ of the sample space can only increase relative entropy: $D_{KL}(PT^{-1}, QT^{-1}) \geq D_{KL}(P, Q)$, with equality iff $T$ is a sufficient statistic for $\{P, Q\}$ and where $PT^{-1}(A) = P(T^{-1}(A))$. That's all I know about increase of entropy and the impossibility to create information.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.