# Variational principles for approximating thermodynamic potentials in the inverse Ising problem: How to go from double to single extremum?

I'm trying to wrap my head around section 2.2.6 (on variational principles) in the following paper (on the inverse Ising problem): https://arxiv.org/abs/1702.01522

Here the authors explain how to use variational principles to approximate the following thermodynamic potentials: $$F(\boldsymbol{J}, \vec{h}) = - \ln{Z(\boldsymbol{J}, \vec{h})},$$

$$S(\boldsymbol{\chi}, \vec{m}) = \min_{\boldsymbol{J}, \vec{h}}\left\{{-\sum_{i} h_i m_i -\sum_{i < j} J_{ij} \chi_{ij} - F(\boldsymbol{J}, \vec{h}) }\right\},$$

$$G(\boldsymbol{J}, \vec{m}) = \max_{\vec{h}}\left\{ { \sum_{i} h_i m_i + F(\boldsymbol{J}, \vec{h})} \right\}.$$

I understand that $$F(\boldsymbol{J}, \vec{h})$$ can be obtained by noting that for a distribution $$q$$,

$$D_{KL}(q||p) = \langle H \rangle_q + \langle \ln \ q \rangle_q + \ln \ Z(\boldsymbol{J}, \vec{h}),$$ so that, using the notation $$U[q] \equiv \langle H \rangle_q$$, and $$S[q] \equiv \langle \ln \ q \rangle_q$$, we have: $$F(\boldsymbol{J}, \vec{h}) = \langle H \rangle_q + \langle \ln \ q \rangle_q - D_{KL}(q||p) \longrightarrow F(\boldsymbol{J}, \vec{h}) = \min_q\{ U[q] - S[q]\}.$$

However, the authors then go on to say that it is easy to see (using Lagrange multipliers) that the remaining two potentials can be expressed as:

$$G(\boldsymbol{J}, \vec{m}) = \max_{\vec{h}}\left\{ { \sum_{i} h_i m_i + \min_q\{ U[q] - S[q]\}} \right\} = \min_{q \in \mathcal{G}}\left\{-\sum_{i < j} J_{ij} \langle\sigma_i \sigma_j \rangle_q - S[q]\right\},$$ and $$S(\boldsymbol{\chi}, \vec{m}) = \min_{\boldsymbol{J}, \vec{h}}\left\{{-\sum_{i} h_i m_i -\sum_{i < j} J_{ij} \chi_{ij} - \min_q\{ U[q] - S[q]\} }\right\} = \max_{q \in \mathcal{S}}\{S[q]\}.$$ Where $$\mathcal{G} \equiv \{q : \langle \sigma_i \rangle_q = m_i\},$$ and $$\mathcal{S} \equiv \{q : \langle \sigma_i \rangle_q = m_i \ \& \ \langle \sigma_i \sigma_j \rangle_q = \chi_{ij} \}$$.

My question is how do I, via Lagrange multipliers, go from these double extremum expressions to the single extremum expressions. Thanks in advance for any help I may recieve.

## 1 Answer

I think I understand how to get the correct expressions now, however, I did not use Lagrange multipliers:

We want to show that

$$S(\boldsymbol{\chi}, \vec{m}) \equiv \min_{\boldsymbol{J}, \vec{h}}\left\{{-\sum_{i} h_i m_i -\sum_{i < j} J_{ij} \chi_{ij} - \min_q\{ U[q] - S[q]\} }\right\},$$ can be expressed as $$S(\boldsymbol{\chi}, \vec{m}) = \max_{q \in \mathcal{S}}\{S[q]\}.$$

Note first that $$-\min_q \{U[q] - S[q]\} = \max_q\{-U[q] + S[q]\}$$. Let

$$S(\boldsymbol{\chi}, \vec{m} ; \boldsymbol{J}, \vec{h}; q) = - \sum_i h_i m_i -\sum_{i < j} J_{ij} \chi_{ij} - U[q] + S[q],$$ then, since $$U[q] \equiv \langle H \rangle_q = \langle -\sum_i h_i \sigma_i - \sum_{i < j} J_{ij} \sigma_i \sigma_j \rangle_q = -\sum_i h_i \langle \sigma_i \rangle_q - \sum_{i < j} J_{ij} \langle \sigma_i \sigma_j \rangle_q,$$ if we restrict the distributions $$q$$ to distributions $$q \in \mathcal{S} \equiv \{ q : \langle \sigma_i \rangle_q = m_i \text{ and } \langle \sigma_i \sigma_j \rangle_q = \chi_{ij} \},$$ we obtain \begin{align*} S(\boldsymbol{\chi}, \vec{m} ; \boldsymbol{J}, \vec{h}; q) &= - \sum_i h_i m_i -\sum_{i < j} J_{ij} \chi_{ij} - \Big( -\sum_i h_i \langle \sigma_i \rangle_q - \sum_{i < j} J_{ij} \langle \sigma_i \sigma_j \rangle_q \Big) + S[q] \\ &= \Big( - \sum_i h_i m_i -\sum_{i < j} J_{ij} \chi_{ij} +\sum_i h_i m_i + \sum_{i < j} J_{ij} \chi_{ij} \Big) + S[q] \\ &= S[q]. \quad (q \in \mathcal{S}) \end{align*} Thus, $$S(\boldsymbol{\chi}, \vec{m} ; \boldsymbol{J}, \vec{h}; q) = S(\boldsymbol{\chi}, \vec{m}; q)$$, i.e., there is no dependecy on $$\boldsymbol{J}$$ and $$\vec{h}$$. To get $$S(\boldsymbol{\chi}, \vec{m})$$ then, we simply re-introduce the maximization over $$q$$ (with our condition that $$q \in \mathcal{S}$$):

$$S(\boldsymbol{\chi}, \vec{m}) = \max_{q \in \mathcal{S}} \{ S[q]\}.$$

A similar argument yields the correct expression for $$G(\boldsymbol{J}, \vec{m})$$.