I have been going through E. T. Jaynes' 1957 paper, Information Theory and Statistical Mechanics. There is a step in his derivations, which has been giving me headaches for the past day; would appreciate some pointers on how to complete it!

Link to paper here: https://bayes.wustl.edu/etj/articles/theory.1.pdf

A bit of notation: Lagrange multipliers are represented by $\lambda$'s, and $\lambda_0$ = ln Z where Z is the partition function. The probability of each state $j$ is

$\begin{equation} p_j = \frac{\exp \left[ -(\lambda_0 + \lambda_1 f_1(x_j) + \cdots \right]}{Z}. \end{equation}$

The expectation values of the functions $f's$ are fixed/known, and provide the constraints for maximizing Shannon's entropy; for simplicity let's consider only $f_1$. Assume that the probabilistic states $i$'s are discrete.

Now onto the question. Jaynes says near Equation (5.1) of the paper that he's going to perturb $f_1$ such that $f_1(x_j) \rightarrow f_1(x_j) + \delta f_1(x_j)$, for all $j$. At the same time, the expectation value of $f_1$ is independently altered: $\left<f_1\right> \rightarrow \left<f_1\right> + \delta\left<f_1\right>$.

How does the entropy, the derived probability distribution, and the Lagrange multiplier change? Equation (5.1) states that

$\begin{equation} \delta\lambda_0 = \delta \mathtt{ln} Z = -\left( \delta\lambda_1 \left<f_1\right> + \lambda_1 \left<\delta f_1\right> \right). \end{equation}$

But how? Here's my current approach:

$ \begin{eqnarray} \delta\lambda_0 &= ln Z' - ln Z \\ &= ln (Z' / Z) \\ &= ln (\frac{\sum_j \exp[(\lambda_1 + \delta\lambda_1)\times (f_1(x_j) + \delta f_1(x_j))]}{Z}). \end{eqnarray} $

But then I am pretty lost on how to "get rid of" the logarithmic function...Thanks for reading, and looking forward to see what you think!

  • $\begingroup$ I do not think the tag "research-level" is appropriate for this question. $\endgroup$ – lattitude Jun 9 at 7:21
  • $\begingroup$ That's fair --- even though I am reading the paper for grad work, the math is admittedly "basic" relative to the curriculum. Tag removed. $\endgroup$ – Eve L Jun 10 at 20:35

Remember that $\delta\lambda_1$ and $\delta f_1$ are small so expanding the exponential and then the log to first order: $$\eqalign{ \delta\lambda_0&\simeq\ln\Big[{1\over{\cal Z}}\sum_je^{-\lambda_1 f_1(x_j)-\delta\lambda_1 f_1(x_j)-\lambda_1\delta f_1(x_j)}\Big]\cr &=\ln\Big[{1\over{\cal Z}}\sum_je^{-\delta\lambda_1 f_1(x_j)-\lambda_1\delta f_1(x_j)}e^{-\lambda_1 f_1(x_j)}\Big]\cr &\simeq \ln\Big[{1\over{\cal Z}}\sum_j\Big(1-\delta\lambda_1 f_1(x_j)-\lambda_1\delta f_1(x_j)\Big)e^{-\lambda_1 f_1(x_j)}\Big]\cr &=\ln\Big(1-\delta\lambda_1\langle f_1\rangle-\lambda_1\langle \delta f_1\rangle\Big)\cr &\simeq -\delta\lambda_1\langle f_1\rangle-\lambda_1\langle \delta f_1\rangle }$$ Alternatively, a faster calculation is $$\eqalign{ \delta \ln{\cal Z}&={1\over{\cal Z}}\delta{\cal Z}\cr &={1\over{\cal Z}}\sum_j\Big(-\delta\lambda_1 f_1(x_j)-\lambda_1\delta f_1(x_j)\Big)e^{-\lambda_1 f_1(x_j)}\cr &=-\delta\lambda_1\langle f_1\rangle-\lambda_1\langle \delta f_1\rangle }$$

  • $\begingroup$ Thank you thank you! My main problem was that I couldn't get the right expansion; it is great to see the steps worked out :) $\endgroup$ – Eve L Jun 10 at 20:41

The equation 5.1 in the Jaynes' paper can be understood from basic knowledge in differential equations. $\delta lnZ$ can be written as

$$\delta \lambda_0 = \delta lnZ = \frac{\delta lnZ}{\delta \lambda_1}\delta \lambda_1 + \sum_{i}\frac{\delta lnZ}{\delta f_1(x_i)}\delta f_i(x_i)$$

Here, we know that the first factor in the first term is the expectation value of the variable corresponding to that Lagrange multiplier. Similarly, the first factor in the summation turns out to be $p_i$, the probability, times $\lambda_1$. Thus,

\begin{equation} \begin{split} \delta \lambda_0 & = -(\delta \lambda_1\langle f_1\rangle + \lambda_1\sum_i p_i \delta f_i(x_i))\\ & =-(\delta \lambda_1\langle f_1\rangle + \lambda_1\langle \delta f_1\rangle) \end{split} \end{equation}

You can derive this expression using your way of taking a difference between the new and old $\mathcal{Z}$, but the same method needs to be used to split the equation and simplify it.

  • $\begingroup$ Thanks for providing a framework for solving the problem! I have to accept 1 of 2 answers, but it doesn't mean I like yours less :P $\endgroup$ – Eve L Jun 10 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.