# Maximizing entropy with Lagrange multipliers

This is a problem I saw in a stat mech textbook, and I think it is a fairly common problem.

Given the entropy function: $$S = - \sum_{i=1}^N p_i \log p_i$$ Maximize $$S$$ subject to constraints:

$$\sum_{i=1}^N p_i = 1 \\ \sum_{i=1}^N p_i e_i = c$$

It was suggested to solve this problem using Lagrange multipliers. So this is how I went about it: $$L(p,\lambda, \mu) = - \sum_{i=1}^N p_i \log p_i - (\lambda \sum_{i=1}^N p_i -1)- (\mu \sum_{i=1}^N p_i e_i - c)$$

$$\frac{\partial L}{\partial p_k} = -(\log p_i +1) - \lambda - \mu e_i = 0$$ A little arithmetic gives:

$$p_i = e^{-\lambda - \mu e_i -1}$$

Then I used the above constraints to solve for $$p_i$$.

$$\sum_{i=1}^N p_i = \frac{\sum e^{-\mu e_i}}{e^{\lambda+1}} = 1 \implies e^{\lambda +1} = \sum e^{-\mu e_i}$$

And

$$\sum_{i=1}^N e_i p_i = \frac{\sum_{i=1}^N e_i e^{-\mu e_i}}{e^{\lambda +1}} = c$$

Since I am not sure how to solve this final constraint and get a value for $$\mu$$, I said

$$p_i = \frac{e^{-\mu e_i}}{\sum_i e^{-\mu e_i}}$$

My question is, how do I solve for $$\mu$$?

## 2 Answers

One may apply a following trick, let $$f(\mu) = \sum e^{-\mu e_i}$$, then: $$\frac{\partial f}{ \partial\mu} = -\sum e_i e^{-\mu e_i}$$ the constraints lead to following differential equation: $$\frac{\partial f}{ \partial\mu} = -cf \qquad f(0) = N$$ Where $$N$$ is number of operands in $$e_i$$. Which has a solution: $$f(\mu) = N e^{-c \mu}$$ However, in general, there is no way to resolve the equation for $$\mu$$ in a closed form: $$\sum e^{-\mu e_i} = N e^{-c \mu}$$ One may get a solution by some numerical technique.

You cannot solve for $$\mu$$ unless you know the $$e_i$$'s. But that shouldn't bother you, because in the context of the canonical ensemble, $$\mu$$ is defined to be the (inverse) temperature, and all quantities are written in terms of it. $$c$$ is the expected energy for a given $$\mu$$.