8
$\begingroup$

In physics, the entropy is related to the volume of microstates associated with a macrostate. So entropy is a property of a macrostate.

However, does it make sense to assign an entropy to a microstate? More importantly, is there any physical significance?

For example, a state in which all the atoms in a gas are distributed in a grid within a box seems a lot more orderly than a distribution in which atoms are distributed randomly. Both distributions can belong the same macrostate (say one defined by the energy) or different macrostates, depending on how you define the macrostate. Each state is equally probable, but the first is easier to describe. There's an analogy in the world of passwords. The password "aaaa" has a lower entropy than the password "ahytr".

$\endgroup$
2
  • 4
    $\begingroup$ No, entropy is a function of our description of the system. In particular, it is not a property of the physical system itself, i.e. the microstate. $\endgroup$ Nov 19, 2022 at 21:36
  • $\begingroup$ Yes, because increase in kinetic energy of a particle increase entropy. The same temperature in one sense increase entropy and in other decrease. A system having same average value for all particles has less entropy while eqiiprobable having different value is more chaotic. $\endgroup$ Nov 19, 2022 at 23:46

3 Answers 3

17
$\begingroup$

When being strict the answer is "no": The entropy is always a property of the ensemble not of the microstates and by definition all microstates in a microcanonical ensemble have the same probability.

The same holds for your password example – a password chosen uniformly from the set of $n$-letter sequences means all passwords are to be equally likely.

If you preclude strings like "aaaa" from your password selection method, you actually make your passwords weaker – because the entropy of your password distribution is reduced. Of course, this does not hold in the real world, because in the real world the attacker will always try a dictionary attack first and "aaaa" will be in the dictionary. (Nevertheless, doing similar things broke encryption algorithms in the past, e.g. the Enigma machine never maps a character to itself under encryption, which massively helped its cryptanalysis).


What you think of as the entropy of a specific microstate or of a string can, however, be made a strict concept, which is called Kolmogorov complexity. Instead of supplying the entropy of the ensemble/distribution you can quantify the "amount of information" by determining the length $K_L(s)$ of the shortest program in a Turing complete language $L$ required to generate the specific string $s$.

This of course depends on the language chosen to encode the string, however, one can prove that given to encoding languages $L_1$ and $L_2$ there exists a constant $c$ only dependent on the languages that bounds the difference of the Kolmogorov complexities of any string with respect to $L_1$ and $L_2$:

An important result about Kolmogorov complexity is, that a string generated by a Markov source, will have almost certainly a Kolmogorov complexity that is equal to the source entropy (as the string length goes to infinity). (In other words, the strings compressible beyond entropy coding are a negligible set among all strings.)

The Kolmogorov complexity, however, is not computable and thus not very useful besides its theoretical implications.


Other than thinking about Kolmogorov complexity, you can also think about the entropy of a specific "microstate" by subdividing your system into smaller, weakly-interacting, equvialent subsystems and re-interpret your system as an ensemble of these subsystems. (This kind of thinking is commonly employed when connecting statistical physics to the thermodynamics of macroscopic systems.)

$\endgroup$
11
$\begingroup$

I would like to add a complement to the excellent Sebastian Riese's answer.

Entropy is an overloaded term. Speaking about entropy without specifying which entropy one is using is the best way to introduce misunderstandings and apparent contradictions.

Let me list a few facts about entropy (or, better, entropies) with an eye to the original question.

  • There is the original thermodynamic (Clausius) entropy. There is no room for microstates in such a case (classical Thermodynamics).
  • Statistical Mechanics introduces a general form for the equilibrium entropy (the generalized Gibbs entropy), valid in each ensemble characterized by a probability distribution of the microstates ($P_i$): $$ S_{Gibbs} = - k_B \sum_i P_i \log P_i. \tag{1} $$ Gibbs' entropy can be considered as the application to the Statistical Mechanics ensembles of the Shannon formula for the Information Theory entropy. By using the expression for $P_i$ relevant for each ensemble, it is a simple exercise to recover the correct expression for the entropy of that ensemble. There are a few observations about Gibbs' expression. Firstly, it is based on the probabilities of microstates and a sum over all the microstates. Therefore it is a property of the macrostate in the specific ensemble. Even worst, the entropy corresponding to formula $(1)$ usually lacks some essential properties of the Thermodynamic entropy (extensiveness, convexity, presence of points non-analyticity corresponding to phase transitions). We can recover such essential properties of the thermodynamic entropy only by taking the so-called Thermodynamic Limit (TL). That is fine, but we face the additional problem that we lose even an analytic expression for the probabilities of the microstates.
  • The only way to assign an entropy to a single microstate is by using the concept of Kolmogorov entropy (also known as algorithmic complexity, Solomonoff–Kolmogorov–Chaitin complexity, program-size complexity, descriptive complexity, or algorithmic entropy). If we could evaluate the algorithmic complexity of each microstate, its average would give us the Shannon-Gibbs entropy. Unfortunately, as noticed by Sebastian Riese, such a concept corresponds to a non-computable function. Moreover, even if it could be evaluated, its value for a single microstate could be quite different from the average value. Said in another way, the algorithmic complexity of one configuration alone would not provide helpful information for the thermodynamic entropy of the macrostate.
$\endgroup$
3
$\begingroup$

For example, a state in which all the atoms in a gas are distributed in a grid within a box seems a lot more orderly than a distribution in which atoms are distributed randomly. Both distributions can belong the same macrostate (say one defined by the energy) or different macrostates, depending on how you define the macrostate. Each state is equally probable, but the first is easier to describe. There's an analogy in the world of passwords. The password "aaaa" has a lower entropy than the password "ahytr".

To take in a slightly different direction: the preferred "orderly" configurations play important role in the theory of critical phenomena (phase transitions). Thus, in disordered phase the "orderly" and "disorderly" configurations can be equally possible microstates, belonging to the same macrostate; however in the ordered phase only the "orderly configuartions are possible. There is typically an entropy change associated with the transition. A related notion is that of the entropy-driven order.

In this context it is worth mentioning correlation functions, which are more convenient for characterizing ordered and disordered phases (e.g., the probability that two spin separated by distance $r$ are aligned in the same direction in Ising model.) In this sense one speak of long-range order or absence thereof.

Finally, there is also use of entropy for disordered systems, although in this case it is more information/Shannon entropy than a thermodynamic one: see here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.