# Softmax Function - Relation to Stat Mech?

Not sure if this post can be answered here on Phys SE, but I'll give it a shot. I've been reading papers on convolutional neural nets recently, and so I am embarking to learn more about how they work. One paper uses a mathematical function called a Softmax function as their activation function in a particular layer (it's a layer that takes input from several shifted/flipped/RGB changed images, runs them through separate networks, and averages the output).

I was very surprised to see one of the forms of the softmax function, given as follows (Wikipedia):

$$\sigma_i = \frac{\exp(-\beta z_i)}{\sum_{j=1}^{K} \exp(-\beta z_j)}, i=1,\dots,K,\mathbb{\ and\ }z_i \in \mathbb{R}$$

To me, as someone who studies physics more regularly than computer science, it's bizarre to me that the form of this function is identical to that of the Boltzmann distribution. Further to my surprise there is a very short section in the Wikipedia article that acknowledges this similarity.

From a machine learning standpoint, we know it's a normalized function (so that's good for node outputs), but I guess my questions are:

• To put it simply... why? Why is this a good activation function in neural nets? The Boltzmann distribution makes sense to me as it is derived fundamentally in a variety of ways of thinking about particles, energy states, energy levels, etc.
• Is there something more fundamental going on here in this equation, perhaps a more general description of "what it does"? I'm comfortable with its meaning/definition in Physics. How can we "show" that this is a good activation function in the context of neural networks?
• Random guess: the Boltzmann distribution maximizes the entropy for a system at given temperature, so maybe the softmax function is useful because it maximizes the entropy of something.
– d_b
Oct 2, 2021 at 5:27
• Related: stats.stackexchange.com/q/66796 Oct 2, 2021 at 5:28

I don't think there's a particularly deep connection to physics. Some properties that make it useful are

• Softmax is a differentiable function, unlike the "actual" max (which means you can use backpropagation with softmax)
• Exponentials map real numbers to positive real numbers. Therefore, you can take an arbitrary real valued score, which pops out of your neural network, and convert it into a quantity you can think of as a probability.
• It is a natural generalization of logistic function, which is the canonical example of a function you can use for classification with one class, so that it can handle multiple classes.
• Softmax is the distribution that maximizes the entropy for multi-class classification type problems.

The last point is, I suppose, why it appears in both machine learning and physics. But the interpretation is quite different in both cases.

• I was thinking quite a bit about the second point last night, and how it relates to negative temperature. Assuming positive energy levels, a temperature being negative can still be normalized and provide a probability that makes sense. I was thinking about differentiability as well. Maybe it's just a weird coincidence. Oct 2, 2021 at 14:27
• @MichaelBurt Exponentials map any real number, positive and negative, to positive numbers. So negative temperatures will still lead to positive numbers that can be interpreted as a probability, for either positive or negative energies. Oct 2, 2021 at 21:11

Boltzmann distribution for discrete states is $$p_i=Ze^{-\beta E_i},$$ where $$p_i$$ is the probability to find the system in this state, $$\beta$$ is the inverse temperature, and $$E_i$$ is the inverse energy. The partition function $$Z=\sum_ie^{-\beta E_i}$$ assures that the distribution is normalized.

So we have the function, which in some domains, such as machine learning, is referred to as softmax. It is a nice differentiable function, which gives a probability of discrete outcomes, which is the main reason for its wide use in machine learning. Other activation functions, constructed via prescription $$p_i=\frac{f_i(x)}{\sum_i f_i(x)},$$ are being used as well, and often have properties more desired from the computational perspective - e.g., softmax saturates, just like sigmoid, which risks slowing down the computation.

• Not sure why you've been downvoted, this seems like a reasonable answer to me. I've never heard it described as a "probability of discrete outcomes". But that makes sense as you input an energy level (which many states may correspond to) and the function provides the probability of that "energy outcome". Oct 2, 2021 at 14:29
• In the wikipedia article $\beta$ is used as a way of changing in the exponential base to something else. In what way does this change the idea that the softmax function provides a probability of discrete outcomes? It just redistributes those probabilities, right? Oct 2, 2021 at 14:31
• @MichaelBurt changing the bade of exponent only renormalizes the energies. However, using a function like reLU, which has non-vanishing derivative and non-symmetric, changes the convergence speed of neural networks. Oct 2, 2021 at 15:03
• @MichaelBurt you may use "activation function" as the search keyword, to find more alternatives. Note also that softmax is usually used only in the last layer of a network, and is easily replaced by several binary nodes. Oct 2, 2021 at 15:06