What are some critiques of Jaynes' approach to statistical mechanics?

Question

Suggested here: What are the justifying foundations of statistical mechanics without appealing to the ergodic hypothesis?

I was wondering about good critiques of Jaynes' approach to statistical mechanics. @Yvan did a good job in pointing out a couple of them, but I would like to have them fleshed out a bit, especially by someone who is not biased towards Jaynes.

As I think Jaynes' thoughts on this matter shifted a bit through the years, let me summarise what I think his position is:

Logically, one must start with two things:

1. There is a microscopic theory of the phenomenon under consideration --- for the moment that can be the existence of a (quantum or classical) hamiltonian formulation, which then ensures the existence of a preferred Louiville form. Thus it makes sense to discuss probabilities of trajectories (ensemble), independently of where those probabilities come from.

2. There exists a viable macroscopic, coarse-grained description, which is only the case if an experiment says so --- the key is what Jaynes sometimes calls "reproducibility". If a phenomenon is not readily reproduced then clearly one has not gained sufficient control over enough variables --- e.g. it was an experimental fact that controlling temperature and volume of a gas was sufficient to determine its pressure.

Then it is logically true that one could develop a quantitative theory/relationship of the macroscopic degrees of freedom or observables, and the claim is that one should set up an ensemble over the microscopic trajectories subject to the constraints of the macroscopic observations and a unique one is chosen by maximising entropy.

With the ensemble in hand, one could then proceed to make predictions about other observables, failure to then observe them means either your microscopic theory is wrong, or the set of variables chosen is not correct, and the circle of Science is complete.

Importantly this makes no reference to ergodicity, and in fact this works out of equilibrium --- equilibrated systems just tends to help with experimental reproducibility. Personally I see it as morally dimensional analysis, writ large.

Yvan pointed out that there is a problem with classical hamiltonian systems because they have an (uncountable) underlying configuration space, and there are technical problems with defining entropy on them. My opinion is that this does not matter, because all physical hamiltonians are really quantum, and those come with a canonical choice. (Field theories need regularisation, both UV and IR as always, to render the number of degrees of freedom finite to be "physical".) However, I'm probably just being naive and I would certainly welcome education on this.

My preferred reference (never published, I think): Where do we stand on maximum entropy?

Answer 1

I think that, for many physicists, one major criticism of this approach is the feeling, justified by the previous successes of reductionism, that statistical mechanics ought to be derivable logically from the underlying microscopic theory (say, classical or quantum mechanics) . Of course, this is impossible, at least without additional assumptions, since the latter describes the evolution of the system but does not tell you anything about the initial conditions...

One main strength (and limitation) of Jaynes' approach is that it applies regardless of the underlying dynamics (at least as a foundation of equilibrium statistical mechanics). The cost is that, seen from this point of view, equilibirum statistical mechanics is not a fundamental theory of physics, but "only" a special case of statistical inference (in particular, it describes our knowledge about a system and not the system itself).

Answer 2

The MAXENT approach works perfectly well, but with the caveat that it can only be justified when the statistical evolution (e.g., convergence to equilibrium) is deterministic. However this is not much of a restriction: we can’t very well do much predictive physics at all without recourse to at least statistically deterministic dynamics. On the other hand, given the recipe for statistical evolution of a system, the question of whether or not MAXENT methods are necessary at all arises, as Grad said:

[MAXENT] has not yet been connected in any way with dynamics. It can only be looked upon as an ad hoc recipe whose accuracy must be empirically determined. Either [the MAXENT] hypothesis is correct, in which case it is unnecessary, or it is incorrect and should not be used.

[Grad, H. “Levels of Description in Statistical Mechanics and Thermodynamics”. In Delaware Seminar in the Foundations of Physics, Bunge, M., ed. Spinger, New York (1967).]

The empirical determination that Grad refers to is basically identical to determining the relevant macrovariables and some idea of their dynamics. In practice these are usually unknown, and the MAXENT hypothesis is technically incorrect, but a good implementation of the principle will successfully approximate the set of relevant macrovariables and their dynamics based on some reasonable Ansatz. So we amend Grad’s objection that MAXENT “is incorrect and should not be used” to “it is only an approximation of the actual dynamics in and should only be used with care”.

Answer 3

The main criticism: The maximum entropy principle works (i.e., gives a correct description of a physically system) if and only if

the knowledge of the observer is of a very special kind, namely consisting precisely of the expectation values of at least those extensive quantities that are important for a thermodynamic description of the system in question, and

the prior is chosen correctly, consistent with the well-known principles of statistical mechanics.

If one gets the prior wrong (e.g., forgets correct Boltzmann counting), the entropy of mixing doesn't come out correctly, eveen though everything else is done as usual.

If one gets the set of macroobservables wrong - e.g., $H^2$ in place of $H$, or only the total energy when in fact a spatially distributed energy distribution is required for an adequate (nonequilibrium) description - then one gets a meaningless theory inconsistent with observation.

Thus, essentially, Jaynes uses as input what should be a result - namely the correct set of relevant variables, and the correct prior to use. It is a ''derivation'' presupposing the facts, and indeed it was presented only almost a century after the birth of statistical mechanics.

---

I discuss the shortcomings of Jaynes' approach to statistical mechanics (and a number of related problems)

in Section 10.7 of my book Classical and Quantum Mechanics via Lie algebras,

and in various articles in my

theoretical physics FAQ:
• (in Chapter A3)
  What about the subjective interpretation of probabilities?
  Incomplete knowledge and statistics
  Entropy and knowledge
  The role of the ergodic hypothesis

• (in Chapter A6)
  Entropy and missing information
  Ignorance in statistical mechanics
  

Answer 4

Shalizi has alot to say on this matter, but I think some of his anti-Jaynes stuff has flaws. On the other hand, Jaynes is not perfect, as captured by some of the other answers here. That being said, as physical agents observing the world through sensation and perception, we are restricted wholly and fundamentally to inference. Our entire body of science is nothing more than (sometimes) well tested inference. That our fundamental physics should arise as inference techniques should be no surprise. Some brave souls are even beginning to show that quantum mechanics follows simply as an inference technique for particular circumstances (http://arxiv.org/abs/1212.0109).

It seems to me that the only thing we need to get all of statistical mechanics is the max-ent method and the fundamental postulate (isolated systems with fixed energy have uniform distributions over states). From there it's not hard to derive the second law in terms of reproducibility, and to attain many other results. Some impressive ones are Arieh Ben Naim's derivation of the Sackur-Tetrode equation (entropy of ideal gas) strictly from information theoretic considerations, and more recently Dewar's application of MaxEnt to non-equilibrium systems (http://iopscience.iop.org/0305-4470/36/3/303), deriving a max entropy production principle that seems to accord with much of modern day discussions in non-equilibrium stat mech (yielding even the fluctuation theorems!). This is exciting stuff!

So I can't really see the fundamental problem with supposing that stat mech (and even all of physics) is actually just inference. In fact, I find it much less meaningful and relevant to suppose that there really is some objective universe out there that we can get a better handle on than by attaining reproducibility (the essence of science), which really just means consistent inference. Such a supposition would only be an inference, anyways ...

Answer 5

Radu Balescu devotes section 16.2.D of his book "Matter Out of Equilibrium" (Imperial College Press) to Jayne's approach. The focus is on non-equilibrium, because Balescu analyzes the different schools of irreversibility. I will copy and paste:

D) Maximum entropy theory
I now very briefly mention an alternative attempt towards a "grand theory" developed by E.T. Jaynes and his coworkers. This approach was originally inspired from Gibbs' philosophy of equilibrium statistical mechanics (Chap 1). In a first step it involves the determination of the "most plausible" initial condition at $t=0$, given a limited amount of information. This is done by maximizing the entropy, taking into account the information about the state of the system at some previous times. If this initial condition is let to evolve according to the Liouville equation (hence at constant entropy), it will not explain irreversibility. One therefore repeats this process of entropy maximization at successive latter times $t_1 < t_2 < \cdots$: an increase of entropy results. But this process, which amounts to changing the law of evolution at the most elementary level, is not acceptable. Besides being quite arbitrary, this evolution law is not transitive: the one-step evolution from $0$ to $t_2$ does not yield the same result as the evolution from $0$ to $t_1 < t_2$ followed by an evolution from $t_1$ to $t_2$. Although some of the difficulties can be cured, the resulting theory lacks a solid foundations; it is, moreover, extremely complex and has not led to any new concrete results.

Answer 6

This seems like an important critique which has not been mentioned yet.

One purported advantage of Jaynes's approach seems to be a simple statistical interpretation:

Many physicists think that the maximum entropy formalism is a straightforward application of Bayesian statistical ideas to statistical mechanics. Some even say that statistical mechanics is just the general Bayesian logic of inductive inference applied to large mechanical systems. This approach identifies thermodynamic entropy with the information-theoretic uncertainty of an (ideal) observer's subjective distribution over a system's microstates.

However, there has been research (by Professor Cosma Rohilla Shalizi, formally of University of Michigan and now at Carnegie Mellon) which claims that trying to use Jaynes's approach to justify such an interpretation leads to the arrow of time being backwards.

I show that this postulate, plus the standard Bayesian procedure for updating probabilities, implies that the entropy of a classical system is monotonically non-increasing on the average -- the Bayesian statistical mechanic's arrow of time points backwards. Avoiding this unphysical conclusion requires rejecting the ordinary equations of motion, or practicing an incoherent form of statistical inference, or rejecting the identification of uncertainty and thermodynamic entropy.

The link to the full article on ArXiv is here: http://arxiv.org/pdf/cond-mat/0410063v2.pdf

Note: I found the link to this article at the end of a review for Wolfram's book; the reviewer disagrees with Jaynes's ideas but believes them to be more intellectually worthy than Wolfram's.

EDIT: I just realized that @Matt Reece mentioned this in a comment above. I had assumed earlier that no one had mentioned it because when I searched for "arrow of time" on this page I found no results. In any case, I think the fact that an improper interpretation of Jaynes's approach supposedly can lead to the arrow of time being backwards is a non-trivial point which wasn't adequately highlighted so far in the comments -- nevertheless priority for mentioning this observation should go to @Matt Reece.