Physics Stack Exchange is a question and answer site for active researchers, academics and students of physics. It only takes a minute to sign up.
Sign up to join this community
I am looking for a derivation of the formula $$S~=~-\Sigma_ip_i \log (p_i).$$ for entropy, from first principles. I only wish to assume the laws of physics, and without involving concepts in information theory. (After all, the concept of entropy and Boltzmann's formula for it is far older than information theory.)
What is a good definition of entropy? What assumptions are needed to arrive at this? What is the justification to maximizing entropy of a system to arrive at thermodynamics?
You have to assume a couple of things, but first you have to analyze the the system in terms of the phase space. Firts divide the phase space of your system in $k$ cells. Let us assume that your system consist in $N$ particles, where $k\ll N$. We're going to say that a microstate of the system is given by the positions of every single particle. Now, we're gonig to say that the mesostate of the system is given by $\{n_{\alpha}(t)\}\equiv(n_1(t),...,n_N(t))$, where $n_i(t)$ is the average number of particles in the $i$-th cell. And the last thing is this: we are gonig to call the number $W(\{n_{\alpha}(t)\})$ as the number of microsates compatible with the mesostate $\{n_{\alpha}(t)\}$.
Now if $P(\{n_{\alpha}(t)\})$ is the probability of finding the system in the mesostate $\{n_{\alpha}(t)\}$, then we will have
$$P(\{n_{\alpha}(t)\})=W(\{n_{\alpha}(t)\})\cdot p$$ where $p $ is the probability of finding the system un a given microstate.
And now, there are 5 hypotheses that Boltzman assumed to be true:
$$p=\frac{1}{k^{N}}$$ 2. The system evolves from mesostates of lower probability to mesostates of higher probability. $$P(\{n_{\alpha}(t+\tau)\})\geq P(\{n_{\alpha}(t)\}),\qquad \forall\tau>0$$ 3. The The thermodynamic equilibrium state corresponds to the most probable mesostate. If $\{\tilde n_{\alpha}(t)\}$is the most probable mesostate, then $P(\{\tilde n_{\alpha}(t)\})>P(\{n_{\alpha}(t)\}),\qquad\forall\alpha$.
4 There is a relationship between the entropy of the system and the probability of mesostates. $$S=S(W\{n_{\alpha}(t)\})$$ That's all what you need, for derive that relation.
The expression
$$ I = \sum_i -p_i\log p_i $$ is a function of probabilities $p_i$ and although it is often called entropy, it is not the thermodynamic entropy of Clausius (that $S$ from thermodynamics defined through $\int dQ/T$). This is not only because of absence of $k_B$, but also because in order to give $I$ value, one must put in probabilities $p_i$.
No probabilities occur in classical thermodynamics, hence it is not possible to derive the above formula from thermodynamic laws.
However, there is a connection between $I$ and thermodynamic entropy $S$. This connection is: if a system is in equilibrium with reservoir so that it has volume $V$ and average of energy is $U$, a statistical estimation of its thermodynamic entropy $S^*$ (a function of $U,V$) can be calculated as the maximum possible value of $k_BI$ for all possible values of $p_i$ under the imposed constraints (volume is fixed to $V$, average of energy is $U$).
This rule was not, as far as I know, falsified for macroscopic bodies for which it is meant to be used. Why it is valid is not immediately clear.
The information theory comes in when we ask: what is the meaning of $I$ for arbitrary values of $p_i$? The answer it gives is: it is a measure of amount of data that is needed to exactly specify the microstate of the system given those probabilities.
With this interpretation of $I$ the connection can be rephrased in this way:
if a system is in equilibrium with reservoir so that it has volume $V$ and average of energy is $U$, the measure of uncertainty $I$ about the exact microstate given the macroscopic constraints $U$,$V$ is the same function of $U,V$ as thermodynamic entropy divided by k_B.
This relation has been verified for rarified gas and other simple cases and it is simply assumed it holds universally for any macroscopic system in thermodynamic equilibrium.
