Let $\mathbb{R}_{\geq 0}$ be the set of real numbers greater or equal to zero. Assume an average value $\overline{R} \in \mathbb{R}$, called the prior. Then, the probability distribution $p(r), \forall r \in \mathbb{R}_{\geq 0}$ which maximizes the entropy based on the prior is the Gibbs distribution. Since $\mathbb{R}$ is an uncountable set, the partition function is an integral:

$$ Z=\int_0^\infty e^{-\beta r}dr=\frac{1}{\beta} $$

The average value $\overline{R}$ is

$$ \overline{R}=-\frac{1}{Z}\frac{\partial Z}{\partial \beta}=-\beta (- \beta^{-2})=\frac{1}{\beta} $$

and the entropy is

$$ S=k_B(\ln [Z]+\beta \overline{R})=k_B\left(\ln[\beta^{-1}]+\beta \frac{1}{\beta}\right)=k_B \left( 1-\ln [\beta] \right) $$

Graphing the entropy yields:

enter image description here

Why is the entropy negative when $1<\ln[\beta]$? Should the entropy not be greater than zero for all values of $\beta$? Error somewhere?


As requested in the comments, the Log Plot of S is

enter image description here

and $\beta$ is a Lagrange multiplier.


As clarified in the comments, here is the plot of $S$ with respect to $\ln{\beta}$.

enter image description here

  • $\begingroup$ Two things: 1) please define $\beta$, and 2) plot the entropy on a log scale. It's weird that you've defined the average to be $\bar{R}$ but then introduce $\beta$ as a separate parameter. $\endgroup$ – DanielSank Mar 2 '19 at 19:51
  • $\begingroup$ @DanielSank I'm assuming $\beta=1/kT$ , at least this is what I have always seen. $\endgroup$ – BioPhysicist Mar 2 '19 at 20:00
  • $\begingroup$ Alexandre, are you looking at a system whose energy varies linearly with respect to some parameter $r$, and where there is no degeneracy for any energies? $\endgroup$ – BioPhysicist Mar 2 '19 at 20:02
  • $\begingroup$ Please make your question one cohesive post. An edit history is available for those who are interested. $\endgroup$ – BioPhysicist Mar 2 '19 at 20:08
  • $\begingroup$ @Aaron correct, it physically corresponds to a system where the energy varies linearly and where there is no degeneracy. One way to think of it is an entropic force $F$ over a distance $x$. In this case $E=Fx$. $\endgroup$ – Alexandre H. Tremblay Mar 2 '19 at 20:13

Look up differential entropy.

For continuous distributions, you have to compute the differential entropy, which is expected to be negative sometimes, and not the Boltzmann entropy or the Shannon entropy or the von Neumann entropy.

The mutual information is always positive, even the mutual information between two continuous distributions where the differential entropy is negative.

| cite | improve this answer | |
  • $\begingroup$ That answer is concerning. Since most of introduction statistical physics is based on the differential entropy, how deep does the error go? For instance, the idea gas law is obtained using the differential entropy. Do both definitions accidentally agree on physically-relevant cases (it seems they would not)? $\endgroup$ – Alexandre H. Tremblay Mar 2 '19 at 22:59
  • 1
    $\begingroup$ Differential entropy and Boltzmann entropy more or less agree, up to an additive normalizing constant. The problem is that, with continuous distributions, there is really no such thing as a microstate (like Boltzmann entropy requires). For any coarse-grained definition of microstates, you can always pick a finer-grained definition one. This is why you end up with differential entropy. $\endgroup$ – Peter Shor Mar 3 '19 at 3:51
  • $\begingroup$ It's perfectly possible to make introductory statistical physics perfectly rigorous, using the techniques that mathematicians and information theorists use to deal with differential entropy. But physicists tend to just sweep these minor inconsistencies under the rug and pretend entropy is always positive, probably because they're not worried about proving theorems but about making their theories agree with experiment. $\endgroup$ – Peter Shor Mar 3 '19 at 3:53
  • $\begingroup$ do you believe it is possible to derive the ideal gas law starting with relative entropy? $\endgroup$ – Alexandre H. Tremblay Mar 3 '19 at 15:00
  • $\begingroup$ @AlexandreH.Tremblay Yes it is, and if you'd like to post a question asking how and link it here, I will be happy to answer. $\endgroup$ – DanielSank Mar 3 '19 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.