# Maximizing mutual information with Restricted Boltzmann Machines and Monte-Carlo sampling

So I've been reading through Koch-Janusz and Ringel's, "Mutual Information, Neural Networks, and the Renormalization Group" (check it out here). I'm currently trying to reimplement some results from their methodology and I'm getting stuck in the details. Specifically, I get that the dependence of their cost function on the parameters of their model seems to cancel out.

Below I give an outline of their Appendix A. You can also check out the original here on pages 7-8.

The authors describe a machine learning algorithm (based on restricted Boltzmann machines) that learns general RG transformations for lattice systems.

This algorithm works by maximizing the real-space mutual information between a given block of visible spins and the hidden spins in the next coarse-grained layer. Intuitively, maximizing the real-space mutual information extracts the long-distance fluctuations (those that are physically relevant). Check out the picture below (again from 1).

Here my recap of their appendix.

Consider a system $$\mathcal{X}=\{x_i,\dots,x_N\}$$ of classical Ising variables $$x_i \in \{0,1\}$$. We divide the system into a "visible" area, $$\mathcal{V}$$, an environment, $$\mathcal{E}$$, and an intermediate buffer, $$\mathcal{B}$$ that separates the two aforementioned parts. Since we care about the long-distance correlations, we ignore $$B$$ in calculating mutual information so as to avoid including short-distance fluctuations. We are given some data distribution $$P(\mathcal{X})=P(\mathcal{V},\mathcal{B},\mathcal{E})$$. $$P(\mathcal{V},\mathcal{E})$$ and $$P(\mathcal{V})$$ are marginalizations over $$P(\mathcal{X})$$, but since marginalizations are compuationally costly, we model these distributions with two standard (contrastive-divergence trained) RBMs, with parameters, $$\Theta$$ and $$\Psi$$, respectively. These distributions we call $$P_\Theta(\mathcal{V},\mathcal{E})$$ and $$P_\Psi(\mathcal{V})$$. Note that this notation is slightly different than the notation used in Koch-Janusz and Ringel.

We define the conditional probability distribution $$P_\Lambda(\mathcal{H}|\mathcal{V})$$ by another RBM parameterized by $$\Lambda$$: \begin{align}% P_\Lambda(\mathcal{H}|\mathcal{V})&=\frac{e^{-E_\Lambda(\mathcal{H}|\mathcal{V})}}{\sum_\mathcal{H}e^{-E_\Lambda(\mathcal{H}|\mathcal{V})}}\\ E_\Lambda(\mathcal{H}|\mathcal{V})&=- \left(\sum_{ij}v_i\lambda_i^jh_j+\sum_ia_iv_i+\sum_jb_jh_j \right). \end{align}%

The goal is to maximize the mutual information between $$\mathcal{E}$$ and $$\mathcal{H}$$, defined: $$$$I_\Lambda(\mathcal{H}:\mathcal{E})=\sum_{\mathcal{H},\mathcal{E}}P_\Lambda(\mathcal{H},\mathcal{E})\log\left(\frac{P_\Lambda(\mathcal{H},\mathcal{E})}{P_\Lambda(\mathcal{H})P(\mathcal{E})}\right).$$$$ Because $$P(\mathcal{E})$$ has no dependence on the parameters, the authors end up maximizing a proxy to the mutual information: $$$$A_\Lambda(\mathcal{H}:\mathcal{E})=\sum_{\mathcal{H},\mathcal{E}}P_\Lambda(\mathcal{H},\mathcal{E})\log\left(\frac{P_\Lambda(\mathcal{H},\mathcal{E})}{P_\Lambda(\mathcal{H})}\right).$$$$

Here, I skip a few steps (you can find them here), but the authors end up with the following equations: $$$$A_\Lambda(\mathcal{H}:\mathcal{E})\approx\sum_{\mathcal{H},\mathcal{E}}P_\Lambda(\mathcal{H},\mathcal{E}) \langle -\Delta E_{\Lambda,\Theta,\Psi}(\mathcal{V},\mathcal{E},\mathcal{H})\rangle_\mathcal{H},$$$$ which they maximize with stochastic gradient descent. Further definitions: $$$$\langle \Delta E_{\Lambda,\Theta,\Psi}(\mathcal{V},\mathcal{E},\mathcal{H})\rangle_\mathcal{H} \equiv \frac{\sum_\mathcal{V} \left(\Delta E_{\Lambda,\Theta,\Psi}(\mathcal{V},\mathcal{E},\mathcal{H})\right) e^{-E_{\Lambda,\Psi}(\mathcal{V},\mathcal{H})}}{\sum_{\mathcal{V}'}e^{-E_{\Lambda,\Psi}(\mathcal{V}',\mathcal{H})}},$$$$ and \begin{align} E_{\Lambda,\Theta}(\mathcal{V},\mathcal{E},\mathcal{H})&= E_\Lambda(\mathcal{V},\mathcal{H})+E_\Theta(\mathcal{V},\mathcal{E})+\sum_j\log[1+\exp(\sum_iv_j\lambda_i^j+b_j)]\\ E_{\Lambda,\Psi}(\mathcal{V},\mathcal{H})&= E_\Lambda(\mathcal{V},\mathcal{H})+E_\Psi(\mathcal{V})+\sum_j\log[1+\exp(\sum_iv_j\lambda_i^j+b_j)]\\ \Delta E_{\Lambda,\Theta,\Psi}(\mathcal{V},\mathcal{E},\mathcal{H})&=E_{\Lambda,\Theta}(\mathcal{V},\mathcal{E},\mathcal{H}) - E_{\Lambda,\Psi}(\mathcal{V},\mathcal{H}) \end{align}

Now, the authors perform two sets of Monte Carlo samples. First, they approximate $$\langle \Delta E_{\Lambda,\Theta,\Psi}(\mathcal{V},\mathcal{E},\mathcal{H})\rangle_\mathcal{H}$$ with a Monte-Carlo average over samples of $$\mathcal{V}$$ (holding not only $$\mathcal{H}$$ but also $$\mathcal{E}$$ fixed). Next, using $$P_\Lambda(\mathcal{H},\mathcal{E})=\sum_{\mathcal{V}'}P_\Lambda(\mathcal{H}|\mathcal{V}')P(\mathcal{V}',\mathcal{E})$$, they replace the sum over $$\mathcal{H}$$ and $$\mathcal{E}$$ with a Monte-Carlo average over samples of $$(\mathcal{V}',\mathcal{E})$$. Then, $$\mathcal{H}=\mathcal{H}(\mathcal{V}')$$ as given by the $$\Lambda$$-RBM.

If you're still here, thanks for sticking through it with me. My question is, where does the $$\Lambda$$ dependence show up?? If you plug in the definition of $$\Delta E$$, you get a bunch of cancellations:

$$$$\Delta E_{\Lambda,\Theta,\Psi}(\mathcal{V},\mathcal{E},\mathcal{H})=E_\Theta(\mathcal{V},\mathcal{E})-E_\Psi(\mathcal{V})$$$$

This obviously has no dependence on $$\Lambda$$. I've been stuck here pretty long so any help would be appreciated. For the original go to page 7 here. Thank you!

• You will most likely get better answers on stats.stackexchange.com, where machine learning rather than physics is the main area of expertise – Nathaniel Mar 24 at 17:36
• Thanks, I'll head over there then. – user1701415 Mar 25 at 7:21