# What are some critiques of Jaynes' approach to statistical mechanics?

I was wondering about good critiques of Jaynes' approach to statistical mechanics. @Yvan did a good job in pointing out a couple of them, but I would like to have them fleshed out a bit, especially by someone who is not biased towards Jaynes.

As I think Jaynes' thoughts on this matter shifted a bit through the years, let me summarise what I think his position is:

1. There is a microscopic theory of the phenomenon under consideration --- for the moment that can be the existence of a (quantum or classical) hamiltonian formulation, which then ensures the existence of a preferred Louiville form. Thus it makes sense to discuss probabilities of trajectories (ensemble), independently of where those probabilities come from.

2. There exists a viable macroscopic, coarse-grained description, which is only the case if an experiment says so --- the key is what Jaynes sometimes calls "reproducibility". If a phenomenon is not readily reproduced then clearly one has not gained sufficient control over enough variables --- e.g. it was an experimental fact that controlling temperature and volume of a gas was sufficient to determine its pressure.

Then it is logically true that one could develop a quantitative theory/relationship of the macroscopic degrees of freedom or observables, and the claim is that one should set up an ensemble over the microscopic trajectories subject to the constraints of the macroscopic observations and a unique one is chosen by maximising entropy.

With the ensemble in hand, one could then proceed to make predictions about other observables, failure to then observe them means either your microscopic theory is wrong, or the set of variables chosen is not correct, and the circle of Science is complete.

Importantly this makes no reference to ergodicity, and in fact this works out of equilibrium --- equilibrated systems just tends to help with experimental reproducibility. Personally I see it as morally dimensional analysis, writ large.

Yvan pointed out that there is a problem with classical hamiltonian systems because they have an (uncountable) underlying configuration space, and there are technical problems with defining entropy on them. My opinion is that this does not matter, because all physical hamiltonians are really quantum, and those come with a canonical choice. (Field theories need regularisation, both UV and IR as always, to render the number of degrees of freedom finite to be "physical".) However, I'm probably just being naive and I would certainly welcome education on this.

My preferred reference (never published, I think): Where do we stand on maximum entropy?

-
+1 for the link to Jaynes's article. I didn't know it and it is wonderfully written. I'm enjoying it tremendously. Thanks! –  José Figueroa-O'Farrill Oct 8 '11 at 18:56
@JoséFigueroa-O'Farrill: I personally find Jaynes' continuous diatribe against detractors of his brand of probability theory annoying; for instance here it seriously dilutes what is otherwise a great set of applications of the methodology, which I believe stands on its own without the abstract dialectic. I can strongly recommend his suggestion of going through the exercise of using the Gibbs Algorithm on the precessing spin --- it really is informative about where the various bits of physics enter into the final result. –  genneth Oct 8 '11 at 19:34
I'd really like to draw attention to his derivation of Kubo relations --- it is surely the best I've ever seen in any textbook, and in fact generalises to the high order, non-linear elements! –  genneth Oct 8 '11 at 19:37
I don't mind the diatribe, which is not to say that I agree with Jaynes. I just like what I've read so far about his article on the maximum entropy. He clearly thought deeply about these issues and has an engaging style, his opinions notwithstanding. –  José Figueroa-O'Farrill Oct 8 '11 at 23:27
@genneth: I think that the reference you give has been published in Maximum Entropy Formalism, R. D. Levine and M. T. (Eds), The MIT Press (1978). –  Yvan Velenik Oct 9 '11 at 10:18

I think that, for many physicists, one major criticism of this approach is the feeling, justified by the previous successes of reductionism, that statistical mechanics ought to be derivable logically from the underlying microscopic theory (say, classical or quantum mechanics) . Of course, this is impossible, at least without additional assumptions, since the latter describes the evolution of the system but does not tell you anything about the initial conditions...

One main strength (and limitation) of Jaynes' approach is that it applies regardless of the underlying dynamics (at least as a foundation of equilibrium statistical mechanics). The cost is that, seen from this point of view, equilibirum statistical mechanics is not a fundamental theory of physics, but "only" a special case of statistical inference (in particular, it describes our knowledge about a system and not the system itself).

-
I really don't get this: "...statistical mechanics ought to be derivable logically from the underlying microscopic theory...". The way I understand it, statistical mechanics is the framework by which we may hope to derive thermodynamics from the underlying microscopic theory. Or am I missing something? –  José Figueroa-O'Farrill Oct 9 '11 at 22:38
@JoséFigueroa-O'Farrill : The only information one retains from the micrscopic theory in Jaynes' approach is the Hamiltonian, basically. In his approach, the precise form of the microscopic dynamics (say, Hamiltonian evolution) does not play any role. In this sense, he does not derive Statistical mechanics from the apriori more fundamental microscopic theory. [to be continued] –  Yvan Velenik Oct 10 '11 at 6:58
[part 2] In particular, the interpretation of probabilities in Jaynes' approach is as a description of the state of knowledge of the observer. One might, in principle, hope to give a more mechanical meaning to these probabilities (that's exactly one the other approaches, based on mixing properties of the dynamics) are trying to achieve. [To be continued] –  Yvan Velenik Oct 10 '11 at 6:59
[part 3] As I said above, this can't work without some further assumptions, because all provable statements will require taking initial conditions outside a set of measure zero w.r.t., say, Liouville measure. But there is no reason a priori to define typicality in terms of the Liouville measure (and this can't follow from the Hamiltonian evolution, as the latter has nothing to say about initial conditions). –  Yvan Velenik Oct 10 '11 at 6:59

The MAXENT approach works perfectly well, but with the caveat that it can only be justified when the statistical evolution (e.g., convergence to equilibrium) is deterministic. However this is not much of a restriction: we can’t very well do much predictive physics at all without recourse to at least statistically deterministic dynamics. On the other hand, given the recipe for statistical evolution of a system, the question of whether or not MAXENT methods are necessary at all arises, as Grad said:

[MAXENT] has not yet been connected in any way with dynamics. It can only be looked upon as an ad hoc recipe whose accuracy must be empirically determined. Either [the MAXENT] hypothesis is correct, in which case it is unnecessary, or it is incorrect and should not be used.

[Grad, H. “Levels of Description in Statistical Mechanics and Thermodynamics”. In Delaware Seminar in the Foundations of Physics, Bunge, M., ed. Spinger, New York (1967).]

The empirical determination that Grad refers to is basically identical to determining the relevant macrovariables and some idea of their dynamics. In practice these are usually unknown, and the MAXENT hypothesis is technically incorrect, but a good implementation of the principle will successfully approximate the set of relevant macrovariables and their dynamics based on some reasonable Ansatz. So we amend Grad’s objection that MAXENT “is incorrect and should not be used” to “it is only an approximation of the actual dynamics in and should only be used with care”.

-
+1; though as a proponent of the method I feel that its inability to be magic is not that serious... –  genneth Oct 9 '11 at 0:41
I am actually a fan of Jaynes' approach to statistical physics, and hold the man's work in high esteem. But I am much less enamored of the whole probability theory catfight (I would like to understand it better). –  S Huntsman Oct 9 '11 at 0:45

I would recommend anything by Cosma Shalizi on this subject, e.g. as a starting point:

Bayes, Anti-Bayes (blog posts)

-
As far as I can tell, he is mostly interested in the issue of Bayesian statistics (i.e. something I don't really understand) vs. something I really don't understand. The only one which makes direct reference to stat. mech is this. His point about boiling water by becoming ignorant is a bit bizarre, and I'm unsure how to respond in the same language. I cannot speak to his credentials on statistics, but I hope it is clear that his knowledge of physics is not sufficient to be considered a serious commentator. –  genneth Oct 9 '11 at 1:31
To put it into context, I feel that Jaynes is a physicist first (he did real physics with quantum optical systems), and I actually feel that his work on statistics is almost a blemish. Perhaps I have simply benefited from his proselytising without knowing the exact intellectual environment in which it was done. But in any case, I want to know what's wrong with the approach as physics, not as statistics --- the key difference is the experimental reproducibility. –  genneth Oct 9 '11 at 1:34
genneth, Cosma Shalizi's post you link includes a link to his arxiv paper: arxiv.org/abs/cond-mat/0410063. This contains a clear statement of his result and section IIB in particular contains arguments about how applying a "maximum entropy" procedure doesn't reproduce the expected results of stat mech. Although he's now a statistician and not a physicist, he does have a PhD in physics and it looks like a serious argument to me. (I haven't read enough Jaynes to say how closely Shalizi's assumptions model Jaynes's ideas, but I think you should at least give his arxiv paper a close read.) –  Matt Reece Oct 9 '11 at 4:19
@Matt Reece : But apparently, Section IIB is only a criticism of Jaynes' elementary approach to the second principle of thermodynamics. It does not seem to give any argument against MaxEnt as a logical foundation for equilibrium statistical mechanics. –  Yvan Velenik Oct 9 '11 at 9:50
@genneth: Yes, but as a satisfactory theoretical framework for "non-equilibrium statistical mechanics" does not exist at the moment, discussing its foundational issues seems less important ;) . (I am not talking about linear response theory and such, obviously.) –  Yvan Velenik Oct 9 '11 at 11:33

The main criticism: The maximum entropy principle works (i.e., gives a correct description of a physically system) if and only if

• the knowledge of the observer is of a very special kind, namely consisting precisely of the expectation values of at least those extensive quantities that are important for a thermodynamic description of the system in question, and
• the prior is chosen correctly, consistent with the well-known principles of statistical mechanics.

If one gets the prior wrong (e.g., forgets correct Boltzmann counting), the entropy of mixing doesn't come out correctly, eveen though everything else is done as usual.

If one gets the set of macroobservables wrong - e.g., $H^2$ in place of $H$, or only the total energy when in fact a spatially distributed energy distribution is required for an adequate (nonequilibrium) description - then one gets a meaningless theory inconsistent with observation.

Thus, essentially, Jaynes uses as input what should be a result - namely the correct set of relevant variables, and the correct prior to use. It is a ''derivation'' presupposing the facts, and indeed it was presented only almost a century after the birth of statistical mechanics.

I discuss the shortcomings of Jaynes' approach to statistical mechanics (and a number of related problems)

and in various articles in my
theoretical physics FAQ:
• (in Chapter A3)
What about the subjective interpretation of probabilities?
Incomplete knowledge and statistics
Entropy and knowledge
The role of the ergodic hypothesis

• (in Chapter A6)
Entropy and missing information
Ignorance in statistical mechanics

• -

Shalizi has alot to say on this matter, but I think some of his anti-Jaynes stuff has flaws. On the other hand, Jaynes is not perfect, as captured by some of the other answers here. That being said, as physical agents observing the world through sensation and perception, we are restricted wholly and fundamentally to inference. Our entire body of science is nothing more than (sometimes) well tested inference. That our fundamental physics should arise as inference techniques should be no surprise. Some brave souls are even beginning to show that quantum mechanics follows simply as an inference technique for particular circumstances (http://arxiv.org/abs/1212.0109).

It seems to me that the only thing we need to get all of statistical mechanics is the max-ent method and the fundamental postulate (isolated systems with fixed energy have uniform distributions over states). From there it's not hard to derive the second law in terms of reproducibility, and to attain many other results. Some impressive ones are Arieh Ben Naim's derivation of the Sackur-Tetrode equation (entropy of ideal gas) strictly from information theoretic considerations, and more recently Dewar's application of MaxEnt to non-equilibrium systems (http://iopscience.iop.org/0305-4470/36/3/303), deriving a max entropy production principle that seems to accord with much of modern day discussions in non-equilibrium stat mech (yielding even the fluctuation theorems!). This is exciting stuff!

So I can't really see the fundamental problem with supposing that stat mech (and even all of physics) is actually just inference. In fact, I find it much less meaningful and relevant to suppose that there really is some objective universe out there that we can get a better handle on than by attaining reproducibility (the essence of science), which really just means consistent inference. Such a supposition would only be an inference, anyways ...

-