Variational principles for approximating thermodynamic potentials in the inverse Ising problem: How to go from double to single extremum? I'm trying to wrap my head around section 2.2.6 (on variational principles) in the following paper (on the inverse Ising problem): https://arxiv.org/abs/1702.01522
Here the authors explain how to use variational principles to approximate the following thermodynamic potentials:
$$ F(\boldsymbol{J}, \vec{h}) = - \ln{Z(\boldsymbol{J}, \vec{h})}, $$
$$ S(\boldsymbol{\chi}, \vec{m}) = \min_{\boldsymbol{J}, \vec{h}}\left\{{-\sum_{i} h_i m_i -\sum_{i < j} J_{ij} \chi_{ij} - F(\boldsymbol{J}, \vec{h}) }\right\}, $$
$$G(\boldsymbol{J}, \vec{m}) = \max_{\vec{h}}\left\{ { \sum_{i} h_i m_i + F(\boldsymbol{J}, \vec{h})} \right\}. $$
I understand that $F(\boldsymbol{J}, \vec{h})$ can be obtained by noting that for a distribution $q$,
$$D_{KL}(q||p) = \langle H \rangle_q + \langle \ln \ q \rangle_q + \ln \ Z(\boldsymbol{J}, \vec{h}),$$
so that, using the notation $U[q] \equiv \langle H \rangle_q$, and $S[q] \equiv \langle \ln \ q \rangle_q$, we have:
$$F(\boldsymbol{J}, \vec{h}) = \langle H \rangle_q + \langle \ln \ q \rangle_q -  D_{KL}(q||p) \longrightarrow F(\boldsymbol{J}, \vec{h}) = \min_q\{ U[q] - S[q]\}.$$
However, the authors then go on to say that it is easy to see (using Lagrange multipliers) that the remaining two potentials can be expressed as:
$$G(\boldsymbol{J}, \vec{m}) = \max_{\vec{h}}\left\{ { \sum_{i} h_i m_i + \min_q\{ U[q] - S[q]\}} \right\} = \min_{q \in \mathcal{G}}\left\{-\sum_{i < j} J_{ij} \langle\sigma_i \sigma_j \rangle_q - S[q]\right\}, $$
and
$$S(\boldsymbol{\chi}, \vec{m}) = \min_{\boldsymbol{J}, \vec{h}}\left\{{-\sum_{i} h_i m_i -\sum_{i < j} J_{ij} \chi_{ij} - \min_q\{ U[q] - S[q]\} }\right\} = \max_{q \in \mathcal{S}}\{S[q]\}.$$
Where $\mathcal{G} \equiv \{q : \langle \sigma_i \rangle_q = m_i\},$ and $\mathcal{S} \equiv \{q : \langle \sigma_i \rangle_q = m_i \ \& \ \langle \sigma_i \sigma_j \rangle_q = \chi_{ij} \}$.
My question is how do I, via Lagrange multipliers, go from these double extremum expressions to the single extremum expressions. Thanks in advance for any help I may recieve.
 A: I think I understand how to get the correct expressions now, however, I did not use Lagrange multipliers:
We want to show that
$$S(\boldsymbol{\chi}, \vec{m}) \equiv \min_{\boldsymbol{J}, \vec{h}}\left\{{-\sum_{i} h_i m_i -\sum_{i < j} J_{ij} \chi_{ij} - \min_q\{ U[q] - S[q]\} }\right\},$$
can be expressed as
$$S(\boldsymbol{\chi}, \vec{m}) = \max_{q \in \mathcal{S}}\{S[q]\}.$$
Note first that $-\min_q \{U[q] - S[q]\} = \max_q\{-U[q] + S[q]\}$. Let
$$S(\boldsymbol{\chi}, \vec{m} ; \boldsymbol{J}, \vec{h}; q) = - \sum_i h_i m_i -\sum_{i < j} J_{ij} \chi_{ij} - U[q] + S[q],$$
then, since
$$U[q] \equiv \langle H \rangle_q = \langle -\sum_i h_i \sigma_i - \sum_{i < j} J_{ij} \sigma_i \sigma_j \rangle_q = -\sum_i h_i \langle \sigma_i \rangle_q - \sum_{i < j} J_{ij} \langle \sigma_i \sigma_j \rangle_q,$$
if we restrict the distributions $q$ to distributions
$$q \in \mathcal{S} \equiv \{ q : \langle \sigma_i \rangle_q = m_i \text{ and } \langle \sigma_i \sigma_j \rangle_q = \chi_{ij} \},$$
we obtain
\begin{align*}
    S(\boldsymbol{\chi}, \vec{m} ; \boldsymbol{J}, \vec{h}; q) &= - \sum_i h_i m_i -\sum_{i < j} J_{ij} \chi_{ij} - \Big( -\sum_i h_i \langle \sigma_i \rangle_q - \sum_{i < j} J_{ij} \langle \sigma_i \sigma_j \rangle_q \Big) + S[q] \\
    &= \Big( - \sum_i h_i m_i -\sum_{i < j} J_{ij} \chi_{ij} +\sum_i h_i m_i + \sum_{i < j} J_{ij} \chi_{ij} \Big) + S[q] \\
    &= S[q]. \quad (q \in \mathcal{S})
\end{align*}
Thus, $S(\boldsymbol{\chi}, \vec{m} ; \boldsymbol{J}, \vec{h}; q) = S(\boldsymbol{\chi}, \vec{m}; q)$, i.e., there is no dependecy on $\boldsymbol{J}$ and $\vec{h}$. To get $S(\boldsymbol{\chi}, \vec{m})$ then, we simply re-introduce the maximization over $q$ (with our condition that $q \in \mathcal{S}$):
$$S(\boldsymbol{\chi}, \vec{m}) = \max_{q \in \mathcal{S}} \{ S[q]\}.$$
A similar argument yields the correct expression for $G(\boldsymbol{J}, \vec{m})$.
