Softmax Function - Relation to Stat Mech?

Question

Not sure if this post can be answered here on Phys SE, but I'll give it a shot. I've been reading papers on convolutional neural nets recently, and so I am embarking to learn more about how they work. One paper uses a mathematical function called a Softmax function as their activation function in a particular layer (it's a layer that takes input from several shifted/flipped/RGB changed images, runs them through separate networks, and averages the output).

I was very surprised to see one of the forms of the softmax function, given as follows (Wikipedia):

$$\sigma_i = \frac{\exp(-\beta z_i)}{\sum_{j=1}^{K} \exp(-\beta z_j)}, i=1,\dots,K,\mathbb{\ and\ }z_i \in \mathbb{R}$$

To me, as someone who studies physics more regularly than computer science, it's bizarre to me that the form of this function is identical to that of the Boltzmann distribution. Further to my surprise there is a very short section in the Wikipedia article that acknowledges this similarity.

From a machine learning standpoint, we know it's a normalized function (so that's good for node outputs), but I guess my questions are:

• To put it simply... why? Why is this a good activation function in neural nets? The Boltzmann distribution makes sense to me as it is derived fundamentally in a variety of ways of thinking about particles, energy states, energy levels, etc.
• Is there something more fundamental going on here in this equation, perhaps a more general description of "what it does"? I'm comfortable with its meaning/definition in Physics. How can we "show" that this is a good activation function in the context of neural networks?

Answer 1

I don't think there's a particularly deep connection to physics. Some properties that make it useful are

• Softmax is a differentiable function, unlike the "actual" max (which means you can use backpropagation with softmax)
• Exponentials map real numbers to positive real numbers. Therefore, you can take an arbitrary real valued score, which pops out of your neural network, and convert it into a quantity you can think of as a probability.
• It is a natural generalization of logistic function, which is the canonical example of a function you can use for classification with one class, so that it can handle multiple classes.
• Softmax is the distribution that maximizes the entropy for multi-class classification type problems.

The last point is, I suppose, why it appears in both machine learning and physics. But the interpretation is quite different in both cases.

Answer 2

Boltzmann distribution for discrete states is $$p_i=Ze^{-\beta E_i},$$ where $p_i$ is the probability to find the system in this state, $\beta$ is the inverse temperature, and $E_i$ is the inverse energy. The partition function $$Z=\sum_ie^{-\beta E_i}$$ assures that the distribution is normalized.

So we have the function, which in some domains, such as machine learning, is referred to as softmax. It is a nice differentiable function, which gives a probability of discrete outcomes, which is the main reason for its wide use in machine learning. Other activation functions, constructed via prescription $$p_i=\frac{f_i(x)}{\sum_i f_i(x)},$$ are being used as well, and often have properties more desired from the computational perspective - e.g., softmax saturates, just like sigmoid, which risks slowing down the computation.