3
$\begingroup$

In physics entropy is usually measured in nats. I wonder is there a possible model of a physical system which has entropy of discrete number of nats?

How particles and degrees of freedom should be arranged so this to happen?

I would be interested in both

  • example of an analog (non-discrete) system

  • example of a quantum system

$\endgroup$
1

3 Answers 3

5
$\begingroup$

The simplest non-discrete system without a doubt is the entropy of a random variable that is uniform on an interval of length $e$, e.g. the interval $[0,e]$, and $0$ outside that interval, so that the probability density is constantly $e^{-1}$. This has an entropy of $1$ nat.

$\endgroup$
2
  • $\begingroup$ I love a good trivial example. Nice! $\endgroup$
    – user1504
    Commented Dec 31, 2013 at 19:32
  • $\begingroup$ @user1504 I've added my own answer. $\endgroup$
    – Anixx
    Commented Feb 6, 2014 at 15:13
4
$\begingroup$

An integer number of nats is not generally speaking a meaningful thing. At least I've never seen such a thing arise in any sensible physical or mathematical situation. They're quite similar to radians in that respect: one rarely if ever finds oneself considering an angle of exactly one radian, but we use them anyway because they simplify the mathematics.

That's not to say you can't contrive an example. For example, Wolfram|Alpha tells me that a discrete three-state system has an entropy of one nat if $$ p_1 = \frac{-p_2 \ln p_2 -p_3 \ln p_3 - 1}{W(-p_2 \ln p_2 -p_3 \ln p_3 - 1)}, $$ where $W$ is the Lambert W function. It's hard to imagine this sort of thing coming up in any situation other than one where we're deliberately trying to construct an entropy of one nat.

This can easily be extended to a quantum example, by considering a two-spin system prepared in state $|00\rangle$ with probability $p_1$ and similarly $p(|01\rangle)=p_2$, $p(|10\rangle)=p_3$ and $p(|11\rangle)=0$, with $p_1\dots p_3$ defined as above. Then it will be in a mixed state with a von Neumann entropy of one nat.

If I think of an easy way to construct a non-discrete example I will update this post, but I guarantee it will feel just as contrived as the discrete and quantum examples. We use nats because they avoid the need for Boltzmann's constant, not because the quantity $1\;\text{nat}$ is a meaningful one.

$\endgroup$
1
  • $\begingroup$ I've added my own answer $\endgroup$
    – Anixx
    Commented Feb 6, 2014 at 15:13
1
$\begingroup$

The answer below is based on an observation by the asker of this question, which can be found here and here, although the context is a bit different. (When I wrote it I wasn't aware that those questions were written by the same person as this question, but still I think it's a useful answer for future reference.)

In flat contradiction to my previous answer, there is a nice natural example where $\frac{1}{2}\,\text{nat}$ arises as an amount of information. It arises as a conditional entropy rather than an entropy, but I think it's still interesting. Of course you can turn it into one nat simply by making two independent copies of the same system. This is a purely mathematical example, but it's simple enough that it's easy to imagine it arising in a physical context.

Let $Q$ be a random variable uniformly distributed on the interval $[0,1]$. Let $X$ be a discrete random variable with values $\{0,1\}$, correlated with $q$ such that $p(x=1|Q=q)=q$. (Or to put it another way, $X$ is a random variable whose probability $q$ is uniformly distributed in $[0,1]$.)

It is clear that the entropy of $X$ is one bit, and the entropy of $Q$ diverges. (Or, if you prefer to use Shannon's continuous entropy, it's equal to one bit.) However, we may also ask the value of the conditional entropy $H(X|Q)$, which (informally speaking) is the expected amount of uncertainty left in the variable $X$ after learning the value of $Q$. It is defined as

$$\int_0^1 p(Q=q)H(X|Q=q),$$

or

$$-\int_0^1 dq (q\log q + (1-q)\log(1-q))\\ = \left[ \frac{1}{2}(-x^2\log(x) + x + (x-1)^2 \log(1-x)) \right]_0^1\\ = \frac{1}{2}\,\text{nat}.$$

Given two identical, independent systems of this form with variables $X_1, Q_1$ and $X_2, Q_2$, the conditional entropy $H(X_1X_2|Q_1Q_2) = 2H(X|Q) = 1\,\text{nat}$.

It is also reasonable to ask the value of the mutual information between the two variables $X$ and $Q$. This is given by $\ln 2 - 1/2\,\,\text{nats}$, or $1-\frac{1}{2}\log_2 e\,\,\text{bits}$.

For a more physical example of a one-nat quantity, see this question by Mark Eichenlaub. I'm currently trying to work out if there's a connection between this physical example and the mathematical one I've just presented.

$\endgroup$
4
  • $\begingroup$ Is not it the same as in my post here: physics.stackexchange.com/questions/98067/… and here physics.stackexchange.com/questions/98152/… ? I made already an answer myself but the moderators deleted it. $\endgroup$
    – Anixx
    Commented May 20, 2014 at 5:52
  • $\begingroup$ Yes, you're right, and in retrospect your post must be where I got it from. (I made notes on it at the time and it seems I forgot to make a note of where I found it - sorry about that.) $\endgroup$
    – N. Virgo
    Commented May 20, 2014 at 13:53
  • $\begingroup$ can you please modify the answer so to make clear the origin? $\endgroup$
    – Anixx
    Commented May 21, 2014 at 1:44
  • $\begingroup$ Done. (I'd have done it before but I was in a hurry.) $\endgroup$
    – N. Virgo
    Commented May 21, 2014 at 5:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.