I'm reading about statistical definition of entropy, which says

$$S=-k_B\sum_ip_i\ln p_i,\tag1$$

where $k_B$ is Boltzmann's constant, and $p_i$ is probability of $i$th state to be occupied. But in classical mechanics there's always an uncountable set of states with given energy, e.g. all states between time $t_0$ and $t_1$ for given initial conditions.

Shouldn't $(1)$ be changed to an integral instead of discrete sum? Why is it never (as it seems) done?

  • 3
    $\begingroup$ To anyone who would answer this, please do not stop at "we need a cell size in phase space". There are few if any places in physics where absolute values of entropy matter; it's almost always a ratio of entropies, in which case any such "cell size" (usually claimed to come from $\hbar$) disappears. $\endgroup$ – DanielSank Feb 3 '16 at 20:35

Let us start with an example, called Langevin paramagnetism, where the magnetic moment is described classically, as a vector in three dimensions. Calling $\vec\mu$ this moment, $\vec B$ the magnetic induction and $\theta$ the angle between $\vec\mu$ and $\vec B$. The probability density of the angle $\theta$ is $\rho(\theta)=\frac{1}{\mathcal Z}\exp(\beta\mu B\cos\theta)$, where $\mathcal Z$ is the partition function and can be seen as a normalization constant. Indeed we have (using $x=\cos\theta$) $$\mathcal Z=\int_0^\pi\rho(\theta)\sin\theta\,\mathrm d\theta =\int_{-1}^1\exp(\beta\mu B x)\mathrm dx=2\frac{\sinh(\beta\mu B)}{\beta\mu B}.$$ The average energy is $$E=-\frac{\partial \ln\cal Z}{\partial \beta}=-\mu B\left(\coth(\beta\mu B)-\frac1{\beta\mu B}\right)$$ The entropy is $$\begin{split}S&=-k_{\text B}\int_0^\pi \rho(\theta)\ln\rho(\theta)\,\sin\theta\,\mathrm d\theta=-k_{\text B}\frac1{\cal Z} \int_{-1}^1\exp(\beta \mu B x)\left(\beta\mu Bx-\ln\mathcal Z\right)\mathrm dx\\ &=k_{\text B}\ln\mathcal Z-k_{\text B}\beta\mu B\left(\coth(\beta\mu B)-\frac{1}{\beta \mu B}\right)=k_{\text B}\ln {\cal Z}+\frac{E}{T}. \end{split}$$ If one defines (as usual) the free energy $F$ as $F=-k_{\text B}T\ln\mathcal Z$, then the last expression is $F=E-TS$. This demonstrates that the entropy formula is valid using the expression $S=-k_{\text B}\int\rho\ln\rho$.

In this model, no underlying discretization is needed, but when one uses the definition of $S$ as an integral, the argument in the logarithm may have a physical unit, which gives a hint that the entropy is defined up to a constant. Moreover, discretizing the values of $\theta$ into segments of width $\pi/N$, we can define $p_n=\rho(n\frac\pi N)\frac\pi N$. The statistical entropy becomes $$S=-k_{\text B}\sum_n p_n\ln p_n=-k_{\text B}\sum_n \rho\left(n\frac\pi N\right)\frac \pi N\left[\ln\rho\left(n\frac\pi N\right)+\ln\frac\pi N\right]$$ This is a Riemann sum plus a number that goes to infinity when $N\to\infty$. The preceding formula becomes $$S=-k_{\text B}\int\rho\ln\rho\;\;-k_{\text B}\lim_{N\to\infty}\ln\frac\pi N.$$ As usually only the variations of entropy are used, the infinite constant is irrelevant and one may forget about it and use the entropy defined by the integral.

  • $\begingroup$ What is $\beta$ here? $\endgroup$ – Ruslan Feb 4 '16 at 8:56
  • 1
    $\begingroup$ $\beta=1/k_{\text B}T$. This is used in almost all textbooks and publications. $\endgroup$ – Tom-Tom Feb 4 '16 at 9:46
  • $\begingroup$ It might be obvious, but how do you get past the last equality sign in the equation after "The entropy is"? I can't seem to figure out why this holds: $$E=k_BT-\frac{\mu B}{k_B}\coth\left(\frac{\mu B}{k_BT}\right)$$ if energy was supposed to be $E=\mu B\cos\theta$ as seen in your definition of $\rho(\theta)$. $\endgroup$ – Ruslan Feb 5 '16 at 19:12
  • $\begingroup$ The energy of a position is $-\mu B \cos\theta$, the average energy is given by $E$ computed just above the formula. The last equality is obtained by remarking that the expression of the average energy $E$ appears in the expression of $S$ (and one uses $k_{\text B}\beta=\frac1T$. I have corrected the sign in the exponentials, this does not change anything in the following. Note the in your comment, the enrgy should be written as $k_{\text B}T-\mu B\coth\left(\beta\mu B\right)$. $\endgroup$ – Tom-Tom Feb 6 '16 at 10:31
  • $\begingroup$ Ah, got it, thanks. I seem to almost understand your answer, but a few things still remain unclear: 1. Why did you choose probability the way you did? Boltzmann distribution seems to have total energy in the exponent, while you only use potential energy. 2. In first expression for entropy, why do you include $\sin\theta$ weight as multiplier of first $\rho$ but not to one inside $\ln$? 3. Did you mean moment instead of momentum? $\endgroup$ – Ruslan Feb 6 '16 at 11:37

The main reason sum of countable set instead of integral over continuous volume is introduced in statistical physics in the context of classical physics is simplification of mathematics needed to make use of the probability theory to explain basics of information theory and derive the very notion of information entropy, given by the $p\ln p$ formula. Or use Lagrange multipliers to get Boltzmann distribution.

It is easier to think and reason about formula that contain sums, and even easier if sums contain finite number of terms, like for probability theory of a die. One can explain what information entropy and Boltzmann distribution is based on probability theory easily that way.

Once the principles of information theory/statistical physics are explained, one can reiterate with more involved integral formalism that is more in line with classical mechanics and phase space.

In quantum theory, old founders decided that probabilities should be ascribed to eigenvalues of an operator instead of to states; consequently, the formula with sums is often even if the states possible form an even higher cardinality set than phase space does.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.