Perhaps this a nonsensical question but hear me out.

I have a random variable $x$ whose moments I can calculate. The first moment $<x>$ is zero and the second $<x^2> = X^2$ is something nonzero. I decided to use a Gaussian distribution function to characterize this i.e \begin{equation} P(x) = e^{\frac{-x^2}{2X^2}} \end{equation}

I was questioned on this choice and now, I want to know how to do this generally. I can calculate a few more moments (they get harder though and I can't calculate them all). I need to know if a) Choosing a Gaussian is justifiable and b) if yes, how do I justify and if not, what is the correct distribution? Are they always, for instance, corrections to the Gaussian or something like that?

  • I should add, this is a physics problem so I'm not looking for a perfect proof, just a rough idea of how to proceed – user1936752 Mar 19 '15 at 4:47
  • No, this is not justified at all. Any probability distribution which is symmetric with respect to $x=0$ (i.e. $P(x)=P(-x)$) satisfies your two conditions, and Gaussian is by no means special without further constraints. – Meng Cheng Mar 19 '15 at 5:00
  • 1
    The way you've written the question, it actually doesn't have any physics in it (which is good), so I wonder if it might be off topic here. If it is we can migrate it to Cross Validated (or maybe Mathematics). – David Z Mar 19 '15 at 5:04
  • @MengCheng so if I gave you some more moments, then how would you use the constraints? I know that technically I need all the moments to uniquely specify a distribution but in this case, I don't have them. Alternatively, can I somehow write a general form for a distribution function that satisfies the constraints I've put? – user1936752 Mar 19 '15 at 5:07
  • @user1936752 The moments have to satisfy certain inequalities. Check out – Meng Cheng Mar 19 '15 at 5:23
up vote 6 down vote accepted

There is a sense in which specifying an unknown probability distribution with known mean and variance by a gaussian with the corresponding mean and variance is the correct choice, and that is when those are the only things you are willing to say about the random variable. The gaussian distribution is the maximum entropy distribution with a given mean and variance.

It is the best probability distribution to use when you literally want to assume nothing else about your variable.

Gaussian Case

Let's say the mean and the variance are literally the only things you know about your variable. Then, we might be interested in which probability distribution best describes the variable. Surely we would want to make as general a choice as possible. This can be formulated. If we want to maximize our ignorance of the probability distribution subject to some constraints, we want to maximize the differential entropy of our distribution subject to those constraints.

In your case, this means we want to maximize:

$$ S = - \int dx \, p(x) \log p(x) $$ subject to the constraints $$ \int dx \, p(x) = 1 \quad \int dx \, x \, p(x) = \mu \quad \int dx\, (x-\mu)^2 p(x) = \sigma^2 $$ The first constraint ensures that we have a proper probability density, and the second and third constraints are your observations about the mean and the variance. We can solve this with Lagrange multipliers:

$$ -\int dx \, p(x) \log p(x) + \lambda_0 \left[ \int dx\, p(x) - 1 \right] + \lambda_1 \left[ \int dx \, x \, p(x) - \mu \right] + \lambda_2 \left[ \int dx \, (x- \mu)^2 p(x) - \sigma^2 \right] $$ Where, taking the variation of $p(x)$ we obtain $$ \int dx\, \left[ -\log p(x) - 1 + \lambda_0 + \lambda_1 x + \lambda_2 (x- \mu)^2 \right] = 0 $$ which is only satisfied if the integrand vanishes everywhere, suggesting: $$ p(x) = \exp \left[ 1 + \lambda_0 + \lambda_1 x + \lambda_2 (x - \mu)^2 \right] $$ The only remaining problem is to determine the $\lambda$s from the constraints, from which we find $$ p(x) = \frac{1}{\sqrt{ 2 \pi \sigma^2 } } \exp \left( - \frac{(x-\mu)^2 }{ 2\sigma^2 } \right) $$ The standard gaussian.


What is really neat is that this generalizes. If you happen to have a set of functions $f_i$ for which you know the expectations under your probability distribution: $$ \langle f_i (x) \rangle = \alpha_i $$ The probability distribution with the largest entropy consistent with those observations is: $$ p(x) \propto \exp \left[\sum_i \lambda_i f_i(x) \right] $$ Where the proportionality constant and all of the $\lambda$s must be fixed to make all of the observed observations true. If you wanted a better distribution for your variable, you could go on to measure other quantities and reform your estimate for the probability distribution in this way.

Boltzmann Distribution

What is really interesting is that the probability distribution with maximum entropy that has as its only constraint a known expectation is the exponential distribution: $$ p(x) \propto \exp \left( \frac{x}{\mu} \right) $$ which you might recognize as the Boltzmann distribution

This is not an accident. ET Jaynes would use this fact to build his principle of maximum entropy, and more generally formulate statistical mechanics in terms of information theory. For an nice introduction, consider his paper Information Theory and Statistical Mechanics [doi] [pdf]

  • Amazingly detailed and useful answer! Can I clarify that in my case, I could, for instance take a general form $P(x) = exp\left[\sum_{i}\lambda_{i}x^{i}\right]$ and simply check it for the first few $i$? Also, is it correct to think of the higher $i$ values as somehow less important (like a perturbation type argument where higher powers matter less) or is that wrong? – user1936752 Mar 19 '15 at 7:04
  • 1
    Yes, the maximum entropy probability distribution consistent with measuring the first few moments would be the first few terms in that sum, the moments then give a set of equations to fix the values of the $\lambda$s. This would be the best estimate for the probability distribution. Well, perhaps not the best, but the least wrong estimate. – alemi Mar 19 '15 at 7:10
  • Sorry, I think I didn't phrase that comment very well. What I meant was if I had to choose between knowing $<x^m>$ and $<x^n>$ with $n>m$, does $<x^m>$ matter more in some way? It is typically easier to do expectation values for lower powers but is it true that the maximum entropy distribution obtained from only $<x>$ and $<x^2>$ is better than that obtained from only $<x^3>$ and $<x^4>$ and if so, why? – user1936752 Mar 19 '15 at 7:33
  • 1
    That is a subtle question. Under some assumptions, the maxent distribution formed from the first n moments should converge to the actual distribution. For details see: Maximum entropy and the problem of moments doi pdf So in some sense, the lower moments are more important, but I don't think its all that cut and dried. If you want any more detail than that, I can't help you, perhaps the math stackexchange could. – alemi Mar 19 '15 at 8:01

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.