How do physicists mathematically define entropy (for the second law of thermodynamics) and how is it related to statistical definitions of entropy?

Even though there are many questions on this site about entropy (such as this one), none I could find was mathematically rigorous or had a complete rigorous answer.

I am looking for precise answers that can be understood by mathematicians.

In mathematical statistics, we have many different definitions of the entropy of (or between) probability distributions. Notable ones are:

  • The $\alpha$-entropy $N(\alpha)$ of a distribution $\rho$ on the integers, which is defined as $$\log \sum_{n \in \mathbb{N}} \rho(n)^\alpha.$$ It can be extended to the entropy of a distribution defined on any separable metric space.
  • The Kullback-Leibler divergence (or relative entropy) $$D_{KL}(P, Q) = \int \log\frac{dP}{dQ} dP.$$

Note that transforms $T$ of the sample space can only increase relative entropy: $D_{KL}(PT^{-1}, QT^{-1}) \geq D_{KL}(P, Q)$, with equality iff $T$ is a sufficient statistic for $\{P, Q\}$ and where $PT^{-1}(A) = P(T^{-1}(A))$. That's all I know about increase of entropy and the impossibility to create information.

  • $\begingroup$ More on the definition of entropy. $\endgroup$
    – Qmechanic
    Jun 17 '17 at 18:56
  • $\begingroup$ See my answer to this question. $\endgroup$ Jun 17 '17 at 18:58
  • $\begingroup$ This answer, elaborating on the definition based on the quantum density operator, is very important in physics too. $\endgroup$
    – user154997
    Jun 17 '17 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.