How do physicists mathematically define entropy (for the second law of thermodynamics) and how is it related to statistical definitions of entropy?

Even though there are many questions on this site about entropy (such as this one), none I could find was mathematically rigorous or had a complete rigorous answer.

I am looking for precise answers that can be understood by mathematicians.

In mathematical statistics, we have many different definitions of the entropy of (or between) probability distributions. Notable ones are:

  • The $\alpha$-entropy $N(\alpha)$ of a distribution $\rho$ on the integers, which is defined as $$\log \sum_{n \in \mathbb{N}} \rho(n)^\alpha.$$ It can be extended to the entropy of a distribution defined on any separable metric space.
  • The Kullback-Leibler divergence (or relative entropy) $$D_{KL}(P, Q) = \int \log\frac{dP}{dQ} dP.$$

Note that transforms $T$ of the sample space can only increase relative entropy: $D_{KL}(PT^{-1}, QT^{-1}) \geq D_{KL}(P, Q)$, with equality iff $T$ is a sufficient statistic for $\{P, Q\}$ and where $PT^{-1}(A) = P(T^{-1}(A))$. That's all I know about increase of entropy and the impossibility to create information.

  • $\begingroup$ More on the definition of entropy. $\endgroup$ – Qmechanic Jun 17 '17 at 18:56
  • $\begingroup$ See my answer to this question. $\endgroup$ – Yvan Velenik Jun 17 '17 at 18:58
  • $\begingroup$ This answer, elaborating on the definition based on the quantum density operator, is very important in physics too. $\endgroup$ – user154997 Jun 17 '17 at 19:07

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.