Let $\mathbb{R}_{\geq 0}$ be the set of real numbers greater or equal to zero. Assume an average value $\overline{R} \in \mathbb{R}$, called the prior. Then, the probability distribution $p(r), \forall r \in \mathbb{R}_{\geq 0}$ which maximizes the entropy based on the prior is the Gibbs distribution. Since $\mathbb{R}$ is an uncountable set, the partition function is an integral:

$$ Z=\int_0^\infty e^{-\beta r}dr=\frac{1}{\beta} $$

The average value $\overline{R}$ is

$$ \overline{R}=-\frac{1}{Z}\frac{\partial Z}{\partial \beta}=-\beta (- \beta^{-2})=\frac{1}{\beta} $$

and the entropy is

$$ S=k_B(\ln [Z]+\beta \overline{R})=k_B\left(\ln[\beta^{-1}]+\beta \frac{1}{\beta}\right)=k_B \left( 1-\ln [\beta] \right) $$

Graphing the entropy yields:

enter image description here

Why is the entropy negative when $1<\ln[\beta]$? Should the entropy not be greater than zero for all values of $\beta$? Error somewhere?


As requested in the comments, the Log Plot of S is

enter image description here

and $\beta$ is a Lagrange multiplier.


As clarified in the comments, here is the plot of $S$ with respect to $\ln{\beta}$.

enter image description here

  • $\begingroup$ Two things: 1) please define $\beta$, and 2) plot the entropy on a log scale. It's weird that you've defined the average to be $\bar{R}$ but then introduce $\beta$ as a separate parameter. $\endgroup$
    – DanielSank
    Commented Mar 2, 2019 at 19:51
  • $\begingroup$ @DanielSank I'm assuming $\beta=1/kT$ , at least this is what I have always seen. $\endgroup$ Commented Mar 2, 2019 at 20:00
  • $\begingroup$ Alexandre, are you looking at a system whose energy varies linearly with respect to some parameter $r$, and where there is no degeneracy for any energies? $\endgroup$ Commented Mar 2, 2019 at 20:02
  • $\begingroup$ Please make your question one cohesive post. An edit history is available for those who are interested. $\endgroup$ Commented Mar 2, 2019 at 20:08
  • $\begingroup$ @Aaron correct, it physically corresponds to a system where the energy varies linearly and where there is no degeneracy. One way to think of it is an entropic force $F$ over a distance $x$. In this case $E=Fx$. $\endgroup$
    – Anon21
    Commented Mar 2, 2019 at 20:13

1 Answer 1


Look up differential entropy.

For continuous distributions, you have to compute the differential entropy, which is expected to be negative sometimes, and not the Boltzmann entropy or the Shannon entropy or the von Neumann entropy.

The mutual information is always positive, even the mutual information between two continuous distributions where the differential entropy is negative.

  • $\begingroup$ That answer is concerning. Since most of introduction statistical physics is based on the differential entropy, how deep does the error go? For instance, the idea gas law is obtained using the differential entropy. Do both definitions accidentally agree on physically-relevant cases (it seems they would not)? $\endgroup$
    – Anon21
    Commented Mar 2, 2019 at 22:59
  • 1
    $\begingroup$ Differential entropy and Boltzmann entropy more or less agree, up to an additive normalizing constant. The problem is that, with continuous distributions, there is really no such thing as a microstate (like Boltzmann entropy requires). For any coarse-grained definition of microstates, you can always pick a finer-grained definition one. This is why you end up with differential entropy. $\endgroup$ Commented Mar 3, 2019 at 3:51
  • $\begingroup$ It's perfectly possible to make introductory statistical physics perfectly rigorous, using the techniques that mathematicians and information theorists use to deal with differential entropy. But physicists tend to just sweep these minor inconsistencies under the rug and pretend entropy is always positive, probably because they're not worried about proving theorems but about making their theories agree with experiment. $\endgroup$ Commented Mar 3, 2019 at 3:53
  • $\begingroup$ do you believe it is possible to derive the ideal gas law starting with relative entropy? $\endgroup$
    – Anon21
    Commented Mar 3, 2019 at 15:00
  • $\begingroup$ @AlexandreH.Tremblay Yes it is, and if you'd like to post a question asking how and link it here, I will be happy to answer. $\endgroup$
    – DanielSank
    Commented Mar 3, 2019 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.