Some background for the question.

I've been trying to understand why so many things have a Gaussian Distribution. There are a lot of questions about this on StackExchange but none of them were answered in sufficient detail to satisfy me.

First, I know often times people model phenomenal as Gaussian when they are not to make the math easier. I am not asking about these. I am wondering why so many phenomena actually are Gaussian or approximately Gaussian.

Second, people often say a Gaussian satisfies maximal entropy because energy is conserved and is quadratic (E=.5*mv^2). However, this only means that velocity distributions for systems in thermal equilibrium are Gaussian. However, many things besides velocity such as human height are also Gaussian in the real world.

Third, the Central Limit Theorem is often put as an explanation. People claim that the sum of independent random variables tends to have a Gaussian Distribution so long as it satisfies certain conditions. I believe they are referring to the Lyapunov and Lindeberg variants of the Central Limit Theorem.

Which brings me to my actual question: Why are the conditions of the Lyapunov and Lindeberg Central Limit Theorem often satisfied in the real world?

  • $\begingroup$ Related (closely, imo:): math.stackexchange.com/questions/2379271/… $\endgroup$ – user207480 Oct 4 '18 at 20:32
  • 1
    $\begingroup$ yes. that is one of the questions I had found unsatisfying answers. they give the CLT as an answer, specifying the lyapunov variant in a comment, but don't elaborate in detail on how the conditions are usually met. they give a convolution argument that seems promising but don't explain what they mean by nice or refer to any theorem to back their argument. $\endgroup$ – roobee Oct 4 '18 at 20:44
  • $\begingroup$ I have to say, although I upvoted it, your question (imo) needs to be more specific or else runs the risk of closure as too general. As in "am wondering why so many phenomena actually are Gaussian or approximately Gaussian."........in clearly defined physical problems, / informal terms, we can say " interactions over time" but that's not much help to you, I know. $\endgroup$ – user207480 Oct 4 '18 at 20:52
  • $\begingroup$ Yeah. I was hoping bringing it down to fulfilling the conditions of Lyapunov and Lindeberg would work but it sounds like that isn't specific enough? Problem is I'm not sure how to specify it even more without the answer being not useful to my actual question. $\endgroup$ – roobee Oct 4 '18 at 21:58
  • $\begingroup$ i have edited my question to hopefully clarify which part is background and which is the specific question i want answered $\endgroup$ – roobee Oct 4 '18 at 22:00

There are many ways of looking at this problem. I think it can be broken up into a few separate but related questions:

  1. Why do many things appear approximately normal? For example human height isn't normally distributed, but it still seems well-approximated by a normal distribution.
  2. Does the CLT actually apply to most real-world distributions?
  3. What is so special about maximum entropy distributions anyway?

I'll try to answer these questions. Let's start with 3. There are different schools of thought in probability; the subjectivist interpretation says that distributions are merely a reflection of our belief or uncertainty in something. If I say I am thinking of an integer between 1 and 10 and ask you to guess what it is, your degree of belief as to what number it is might be, for example, a uniform distribution. Given that you don't know anything else about it, the uniform distribution would be the best choice (it would give you the highest probability of being right). However, you might exploit the fact that humans aren't good random number generators, and so a distribution that weighs 5 or 6 higher than 1 might give you a better chance of winning. The point to all this is that distributions (according to subjectivism) represent your degree of belief, and nothing more.

As it so happens, the distribution with the maximum entropy given your knowledge about something is the one that gives you the highest probability of being right. And if you know nothing else about some measurement/number except that it follows these rules:

  • It is a real number.
  • It is not bounded from below or above.
  • It has an average value.
  • It has a certain degree of variance around this average value.

Then the distribution that gives you the highest chance of being right is the normal distribution. A lot of the time, you can even relax some of these assumptions. For example, if it's bounded, but the bounds are much larger than the variance (human height and weight would be good examples here), then the resulting maximum entropy distribution might be a truncated normal distribution, which looks pretty close to a normal distribution.

Which brings us to question 1. Why do so many distributions that aren't normal still appear approximately normal? For example, the Cauchy, Weibull, Student's t, and log-normal distributions all look approximately normal, for certain parameter values.

The best explanation I can think of for this is that they are all the maximum entropy distribution given some constraints, and the constraints are often similar to the constraints that give the normal distribution, so their shapes can wind up being similar to the normal distribution.

Note that just these few distributions together cover a huge range of systems. For example, the CLT says that if you have a bunch of different independent random variables and you sum them, you get something that looks normal. But if instead of adding, you multiply them, you get the log-normal! Which itself looks 'pretty close' to normal, especially if you only had a few sample measurements to go on.

Now finally, question 2. As others have pointed out, the CLT does seem to apply to a lot of real-world problems, but that's actually not that critical. Even if some of the assumptions of the CLT are relaxed a bit, the resulting distributions still come out looking normal. For example, even if the random variables that are summed are not completely independent, the sum can still come out looking normal, as long as there is some degree of randomness or uncertainty in the random variables. This is certainly the case for human height - the genes controlling height all have very inter-related effects. But the result still comes out looking very close to normal.

| cite | improve this answer | |
  • $\begingroup$ "the CLT says that if you have a bunch of different independent random variables and you sum them, you get something that looks normal". That is actually one of the major points I was confused on. The closest variant of the CLT I could find that says something like that were the Lyapunov and Lindeberg variants. But they have complicated conditions that I am not sure why would often be satisfied in real-life. $\endgroup$ – roobee Oct 6 '18 at 21:45
  • $\begingroup$ you say the CLT variant you linked shows that sums of mostly independent random variables with some uncertainty are also mostly normal. An explanation on why the mathematical conditions in the CLT variant you link approximately translate to -mostly independent random variables with some uncertainty - would also suffice if it would be clearer than an explanation of Lyapunov or Lindeberg conditions. $\endgroup$ – roobee Oct 6 '18 at 22:19
  • $\begingroup$ Lyapunov's condition shouldn't be unusual or surprising, it is merely to establish that the variance of the sum doesn't diverge. It is easy to see how in an approximate real-world scenario, with large but finite $N$, this condition would be satisfied. $\endgroup$ – Al Nejati Oct 6 '18 at 22:57
  • $\begingroup$ What I wanted to get at with my answer though was that satisfying those conditions exactly isn't even critical - lots of systems that violate the conditions for the CLT in various ways still look close to normal, for the reasons I described. $\endgroup$ – Al Nejati Oct 6 '18 at 23:09
  • $\begingroup$ can you explain why if the variance of the sum doesn't diverge than lyapunov's condition is satisfied? $\endgroup$ – roobee Oct 10 '18 at 4:36

There are two reasons that come to mind: firstly, many real-world phenomena are collective actions with many, many steps involved. Brownian motion is a good example, where thousands of collisions with a small particle can result in its random jiggles seen in a microscope. The familiar pin-board sorting of balls into a Pascal's-triangle distribution also has rank after rank of identical disturbances (random bounces), and approximates well a Gaussian distribution.

The second reason is more subtle: we make measurements with instruments that we calibrate, and calibration takes out the zero-offset, and the linear-with-independent-variable terms in the errors, but retains the second and higher moments (sigma-squared and such); when we consider only small errors, the error distribution dominated by the lowest nonvanishing moment, the second moment, is one of those things we always assume is Gaussian. The fact is, in the limit of small random errors, when only the second moment matters, the Gaussian distribution result is identical to the 'real' result of ANY other distribution.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.