Suggested here: What are the justifying foundations of statistical mechanics without appealing to the ergodic hypothesis?

I was wondering about good critiques of Jaynes' approach to statistical mechanics. @Yvan did a good job in pointing out a couple of them, but I would like to have them fleshed out a bit, especially by someone who is not biased towards Jaynes.

As I think Jaynes' thoughts on this matter shifted a bit through the years, let me summarise what I think his position is:

Logically, one must start with two things:

  1. There is a microscopic theory of the phenomenon under consideration --- for the moment that can be the existence of a (quantum or classical) hamiltonian formulation, which then ensures the existence of a preferred Louiville form. Thus it makes sense to discuss probabilities of trajectories (ensemble), independently of where those probabilities come from.

  2. There exists a viable macroscopic, coarse-grained description, which is only the case if an experiment says so --- the key is what Jaynes sometimes calls "reproducibility". If a phenomenon is not readily reproduced then clearly one has not gained sufficient control over enough variables --- e.g. it was an experimental fact that controlling temperature and volume of a gas was sufficient to determine its pressure.

Then it is logically true that one could develop a quantitative theory/relationship of the macroscopic degrees of freedom or observables, and the claim is that one should set up an ensemble over the microscopic trajectories subject to the constraints of the macroscopic observations and a unique one is chosen by maximising entropy.

With the ensemble in hand, one could then proceed to make predictions about other observables, failure to then observe them means either your microscopic theory is wrong, or the set of variables chosen is not correct, and the circle of Science is complete.

Importantly this makes no reference to ergodicity, and in fact this works out of equilibrium --- equilibrated systems just tends to help with experimental reproducibility. Personally I see it as morally dimensional analysis, writ large.

Yvan pointed out that there is a problem with classical hamiltonian systems because they have an (uncountable) underlying configuration space, and there are technical problems with defining entropy on them. My opinion is that this does not matter, because all physical hamiltonians are really quantum, and those come with a canonical choice. (Field theories need regularisation, both UV and IR as always, to render the number of degrees of freedom finite to be "physical".) However, I'm probably just being naive and I would certainly welcome education on this.

My preferred reference (never published, I think): Where do we stand on maximum entropy?

  • 2
    $\begingroup$ +1 for the link to Jaynes's article. I didn't know it and it is wonderfully written. I'm enjoying it tremendously. Thanks! $\endgroup$ Commented Oct 8, 2011 at 18:56
  • 1
    $\begingroup$ @JoséFigueroa-O'Farrill: I personally find Jaynes' continuous diatribe against detractors of his brand of probability theory annoying; for instance here it seriously dilutes what is otherwise a great set of applications of the methodology, which I believe stands on its own without the abstract dialectic. I can strongly recommend his suggestion of going through the exercise of using the Gibbs Algorithm on the precessing spin --- it really is informative about where the various bits of physics enter into the final result. $\endgroup$
    – genneth
    Commented Oct 8, 2011 at 19:34
  • $\begingroup$ I'd really like to draw attention to his derivation of Kubo relations --- it is surely the best I've ever seen in any textbook, and in fact generalises to the high order, non-linear elements! $\endgroup$
    – genneth
    Commented Oct 8, 2011 at 19:37
  • $\begingroup$ I don't mind the diatribe, which is not to say that I agree with Jaynes. I just like what I've read so far about his article on the maximum entropy. He clearly thought deeply about these issues and has an engaging style, his opinions notwithstanding. $\endgroup$ Commented Oct 8, 2011 at 23:27
  • 1
    $\begingroup$ @genneth: I think that the reference you give has been published in Maximum Entropy Formalism, R. D. Levine and M. T. (Eds), The MIT Press (1978). $\endgroup$ Commented Oct 9, 2011 at 10:18

6 Answers 6


I think that, for many physicists, one major criticism of this approach is the feeling, justified by the previous successes of reductionism, that statistical mechanics ought to be derivable logically from the underlying microscopic theory (say, classical or quantum mechanics) . Of course, this is impossible, at least without additional assumptions, since the latter describes the evolution of the system but does not tell you anything about the initial conditions...

One main strength (and limitation) of Jaynes' approach is that it applies regardless of the underlying dynamics (at least as a foundation of equilibrium statistical mechanics). The cost is that, seen from this point of view, equilibirum statistical mechanics is not a fundamental theory of physics, but "only" a special case of statistical inference (in particular, it describes our knowledge about a system and not the system itself).

  • 5
    $\begingroup$ I really don't get this: "...statistical mechanics ought to be derivable logically from the underlying microscopic theory...". The way I understand it, statistical mechanics is the framework by which we may hope to derive thermodynamics from the underlying microscopic theory. Or am I missing something? $\endgroup$ Commented Oct 9, 2011 at 22:38
  • $\begingroup$ @JoséFigueroa-O'Farrill : The only information one retains from the micrscopic theory in Jaynes' approach is the Hamiltonian, basically. In his approach, the precise form of the microscopic dynamics (say, Hamiltonian evolution) does not play any role. In this sense, he does not derive Statistical mechanics from the apriori more fundamental microscopic theory. [to be continued] $\endgroup$ Commented Oct 10, 2011 at 6:58
  • $\begingroup$ [part 2] In particular, the interpretation of probabilities in Jaynes' approach is as a description of the state of knowledge of the observer. One might, in principle, hope to give a more mechanical meaning to these probabilities (that's exactly one the other approaches, based on mixing properties of the dynamics) are trying to achieve. [To be continued] $\endgroup$ Commented Oct 10, 2011 at 6:59
  • $\begingroup$ [part 3] As I said above, this can't work without some further assumptions, because all provable statements will require taking initial conditions outside a set of measure zero w.r.t., say, Liouville measure. But there is no reason a priori to define typicality in terms of the Liouville measure (and this can't follow from the Hamiltonian evolution, as the latter has nothing to say about initial conditions). $\endgroup$ Commented Oct 10, 2011 at 6:59
  • $\begingroup$ To me it seems that Jaynes is one of the only few who acknowledges that, as far as probabilities are concerned, it is not possible to obtain a probability from a purely frequentist approach. As a consequence, any attempt to justify the use of probabilities ab nihilo from pure mechanics is doomed to fail. Even modern ergodic theory has to acknowledge that in the fact that it ascribes to non-ergodic initial conditions an invariant probability measure equal to zero. Now, as you say who has ever said that any preparation protocol would agree with such a statement? $\endgroup$
    – gatsu
    Commented Sep 20, 2016 at 5:57

The MAXENT approach works perfectly well, but with the caveat that it can only be justified when the statistical evolution (e.g., convergence to equilibrium) is deterministic. However this is not much of a restriction: we can’t very well do much predictive physics at all without recourse to at least statistically deterministic dynamics. On the other hand, given the recipe for statistical evolution of a system, the question of whether or not MAXENT methods are necessary at all arises, as Grad said:

[MAXENT] has not yet been connected in any way with dynamics. It can only be looked upon as an ad hoc recipe whose accuracy must be empirically determined. Either [the MAXENT] hypothesis is correct, in which case it is unnecessary, or it is incorrect and should not be used.

[Grad, H. “Levels of Description in Statistical Mechanics and Thermodynamics”. In Delaware Seminar in the Foundations of Physics, Bunge, M., ed. Spinger, New York (1967).]

The empirical determination that Grad refers to is basically identical to determining the relevant macrovariables and some idea of their dynamics. In practice these are usually unknown, and the MAXENT hypothesis is technically incorrect, but a good implementation of the principle will successfully approximate the set of relevant macrovariables and their dynamics based on some reasonable Ansatz. So we amend Grad’s objection that MAXENT “is incorrect and should not be used” to “it is only an approximation of the actual dynamics in and should only be used with care”.

  • 3
    $\begingroup$ +1; though as a proponent of the method I feel that its inability to be magic is not that serious... $\endgroup$
    – genneth
    Commented Oct 9, 2011 at 0:41
  • 2
    $\begingroup$ I am actually a fan of Jaynes' approach to statistical physics, and hold the man's work in high esteem. But I am much less enamored of the whole probability theory catfight (I would like to understand it better). $\endgroup$
    – S Huntsman
    Commented Oct 9, 2011 at 0:45

The main criticism: The maximum entropy principle works (i.e., gives a correct description of a physically system) if and only if

  • the knowledge of the observer is of a very special kind, namely consisting precisely of the expectation values of at least those extensive quantities that are important for a thermodynamic description of the system in question, and
  • the prior is chosen correctly, consistent with the well-known principles of statistical mechanics.

    If one gets the prior wrong (e.g., forgets correct Boltzmann counting), the entropy of mixing doesn't come out correctly, eveen though everything else is done as usual.

    If one gets the set of macroobservables wrong - e.g., $H^2$ in place of $H$, or only the total energy when in fact a spatially distributed energy distribution is required for an adequate (nonequilibrium) description - then one gets a meaningless theory inconsistent with observation.

    Thus, essentially, Jaynes uses as input what should be a result - namely the correct set of relevant variables, and the correct prior to use. It is a ''derivation'' presupposing the facts, and indeed it was presented only almost a century after the birth of statistical mechanics.

    I discuss the shortcomings of Jaynes' approach to statistical mechanics (and a number of related problems)

    and in various articles in my
      theoretical physics FAQ:
    • (in Chapter A3)
      What about the subjective interpretation of probabilities?
      Incomplete knowledge and statistics
      Entropy and knowledge
      The role of the ergodic hypothesis

    • (in Chapter A6)
      Entropy and missing information
      Ignorance in statistical mechanics

  • $\endgroup$

    Shalizi has alot to say on this matter, but I think some of his anti-Jaynes stuff has flaws. On the other hand, Jaynes is not perfect, as captured by some of the other answers here. That being said, as physical agents observing the world through sensation and perception, we are restricted wholly and fundamentally to inference. Our entire body of science is nothing more than (sometimes) well tested inference. That our fundamental physics should arise as inference techniques should be no surprise. Some brave souls are even beginning to show that quantum mechanics follows simply as an inference technique for particular circumstances (http://arxiv.org/abs/1212.0109).

    It seems to me that the only thing we need to get all of statistical mechanics is the max-ent method and the fundamental postulate (isolated systems with fixed energy have uniform distributions over states). From there it's not hard to derive the second law in terms of reproducibility, and to attain many other results. Some impressive ones are Arieh Ben Naim's derivation of the Sackur-Tetrode equation (entropy of ideal gas) strictly from information theoretic considerations, and more recently Dewar's application of MaxEnt to non-equilibrium systems (http://iopscience.iop.org/0305-4470/36/3/303), deriving a max entropy production principle that seems to accord with much of modern day discussions in non-equilibrium stat mech (yielding even the fluctuation theorems!). This is exciting stuff!

    So I can't really see the fundamental problem with supposing that stat mech (and even all of physics) is actually just inference. In fact, I find it much less meaningful and relevant to suppose that there really is some objective universe out there that we can get a better handle on than by attaining reproducibility (the essence of science), which really just means consistent inference. Such a supposition would only be an inference, anyways ...


    Radu Balescu devotes section 16.2.D of his book "Matter Out of Equilibrium" (Imperial College Press) to Jayne's approach. The focus is on non-equilibrium, because Balescu analyzes the different schools of irreversibility. I will copy and paste:

    D) Maximum entropy theory
    I now very briefly mention an alternative attempt towards a "grand theory" developed by E.T. Jaynes and his coworkers. This approach was originally inspired from Gibbs' philosophy of equilibrium statistical mechanics (Chap 1). In a first step it involves the determination of the "most plausible" initial condition at $t=0$, given a limited amount of information. This is done by maximizing the entropy, taking into account the information about the state of the system at some previous times. If this initial condition is let to evolve according to the Liouville equation (hence at constant entropy), it will not explain irreversibility. One therefore repeats this process of entropy maximization at successive latter times $t_1 < t_2 < \cdots$: an increase of entropy results. But this process, which amounts to changing the law of evolution at the most elementary level, is not acceptable. Besides being quite arbitrary, this evolution law is not transitive: the one-step evolution from $0$ to $t_2$ does not yield the same result as the evolution from $0$ to $t_1 < t_2$ followed by an evolution from $t_1$ to $t_2$. Although some of the difficulties can be cured, the resulting theory lacks a solid foundations; it is, moreover, extremely complex and has not led to any new concrete results.


    This seems like an important critique which has not been mentioned yet.

    One purported advantage of Jaynes's approach seems to be a simple statistical interpretation:

    Many physicists think that the maximum entropy formalism is a straightforward application of Bayesian statistical ideas to statistical mechanics. Some even say that statistical mechanics is just the general Bayesian logic of inductive inference applied to large mechanical systems. This approach identifies thermodynamic entropy with the information-theoretic uncertainty of an (ideal) observer's subjective distribution over a system's microstates.

    However, there has been research (by Professor Cosma Rohilla Shalizi, formally of University of Michigan and now at Carnegie Mellon) which claims that trying to use Jaynes's approach to justify such an interpretation leads to the arrow of time being backwards.

    I show that this postulate, plus the standard Bayesian procedure for updating probabilities, implies that the entropy of a classical system is monotonically non-increasing on the average -- the Bayesian statistical mechanic's arrow of time points backwards. Avoiding this unphysical conclusion requires rejecting the ordinary equations of motion, or practicing an incoherent form of statistical inference, or rejecting the identification of uncertainty and thermodynamic entropy.

    The link to the full article on ArXiv is here: http://arxiv.org/pdf/cond-mat/0410063v2.pdf

    Note: I found the link to this article at the end of a review for Wolfram's book; the reviewer disagrees with Jaynes's ideas but believes them to be more intellectually worthy than Wolfram's.

    EDIT: I just realized that @Matt Reece mentioned this in a comment above. I had assumed earlier that no one had mentioned it because when I searched for "arrow of time" on this page I found no results. In any case, I think the fact that an improper interpretation of Jaynes's approach supposedly can lead to the arrow of time being backwards is a non-trivial point which wasn't adequately highlighted so far in the comments -- nevertheless priority for mentioning this observation should go to @Matt Reece.

    • 1
      $\begingroup$ The article in question is over 10 years old and unpublished. It has a single citation - arxiv.org/abs/1508.02421 - which explains (p34) why its results aren't significant. $\endgroup$
      – innisfree
      Commented Sep 26, 2016 at 5:51
    • $\begingroup$ @innisfree Thank you for the reference. Honestly I don't think I know enough to comment intelligently. The paper says that the cross-entropy will increase for open systems in Jaynes's formalism when the updating rule is modified, whereas in a closed system it remains constant -- does that imply that the universe would be an open system? That might be possible after all, although clearly I definitely do not know one way or another. Anyway it does seem to address the issue mentioned in that paper. $\endgroup$ Commented Sep 26, 2016 at 7:20

    Your Answer

    By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

    Not the answer you're looking for? Browse other questions tagged or ask your own question.