Take the 2-minute tour ×
Physics Stack Exchange is a question and answer site for active researchers, academics and students of physics. It's 100% free, no registration required.

I am looking for a derivation of the formula $$S~=~-\Sigma_ip_i \log (p_i).$$ for entropy, from first principles. I only wish to assume the laws of physics, and without involving concepts in information theory. (After all, the concept of entropy and Boltzmann's formula for it is far older than information theory.)

What is a good definition of entropy? What assumptions are needed to arrive at this? What is the justification to maximizing entropy of a system to arrive at thermodynamics?

share|improve this question
@Prathyush if you are not interested in information theory then you are not interested in finding the answer to your question!! The first-principles justification for maximising entropy (and the justification for the formula you give) is that we are clumsy monkeys doing the best we can with our limited experimental resolution: it has little to do with fundamental properties or laws of the universe. If you read the other thread you would find that this was all explained by Jaynes in the 50s, e.g. see the original paper here. It's a must read. –  Mark Mitchison Nov 20 '12 at 5:20
Concerning the flag: this is not one of my strengths, but I read the proposed duplicate as asking for a physics based explanation just like this one. I'm not sure that it got the answers you want, but anyone who has a good answer to this question could just as well post it on the other. –  dmckee Nov 24 '12 at 3:05
@dmckee is right that just because the other question doesn't have the answers you want, that doesn't mean it isn't a duplicate. That being said, I think the other question may be asking for any explanation, it wasn't specific about avoiding information theory. What would help is if you add a link to the other question in this one, of the form "I looked at this but it doesn't answer my question because I want to avoid referencing information theory" or something like that (you don't have to use those exact words of course). –  David Z Nov 24 '12 at 17:07
@DavidZaslavsky This thread does ask for an explanation for entropy, but Its starting point different altogether, It focuses on a derivation "From probability theory for general system." I am looking for something that only based on the principles of physics avoiding information theory or atleast demonstrating the clear application of the principles of information in this context starting from principles of physics. –  Prathyush Nov 25 '12 at 18:04
OK, but don't tell me, put it in the question. –  David Z Nov 25 '12 at 21:11

1 Answer 1

You have to assume a couple of things, but first you have to analyze the the system in terms of the phase space. Firts divide the phase space of your system in $k$ cells. Let us assume that your system consist in $N$ particles, where $k\ll N$. We're going to say that a microstate of the system is given by the positions of every single particle. Now, we're gonig to say that the mesostate of the system is given by $\{n_{\alpha}(t)\}\equiv(n_1(t),...,n_N(t))$, where $n_i(t)$ is the average number of particles in the $i$-th cell. And the last thing is this: we are gonig to call the number $W(\{n_{\alpha}(t)\})$ as the number of microsates compatible with the mesostate $\{n_{\alpha}(t)\}$.

Now if $P(\{n_{\alpha}(t)\})$ is the probability of finding the system in the mesostate $\{n_{\alpha}(t)\}$, then we will have

$$P(\{n_{\alpha}(t)\})=W(\{n_{\alpha}(t)\})\cdot p$$ where $p $ is the probability of finding the system un a given microstate.

And now, there are 5 hypotheses that Boltzman assumed to be true:

  1. All the microstates are equally likely, i.e.

$$p=\frac{1}{k^{N}}$$ 2. The system evolves from mesostates of lower probability to mesostates of higher probability. $$P(\{n_{\alpha}(t+\tau)\})\geq P(\{n_{\alpha}(t)\}),\qquad \forall\tau>0$$ 3. The The thermodynamic equilibrium state corresponds to the most probable mesostate. If $\{\tilde n_{\alpha}(t)\}$is the most probable mesostate, then $P(\{\tilde n_{\alpha}(t)\})>P(\{n_{\alpha}(t)\}),\qquad\forall\alpha$.

4 There is a relationship between the entropy of the system and the probability of mesostates. $$S=S(W\{n_{\alpha}(t)\})$$ That's all what you need, for derive that relation.

share|improve this answer
I did not understand the last point, Please can you elaborate. What is the entropy defined as in the first place. I have modified the question a little bit. –  Prathyush Nov 20 '12 at 4:59
@Prathyush try googling 'Boltzmann's tombstone' ;) –  Mark Mitchison Nov 20 '12 at 5:23
Entropy –  Anuar Nov 20 '12 at 5:59

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.