1
$\begingroup$

Why is a 5 sigma confidence level enough to solidify an experimental result as not being a statistical fluke in particle physics? Other branches of science have different sigma values for which they accept a result. Shouldn’t there be a universal sigma value?

$\endgroup$
3
$\begingroup$

First, to be clear, physics, with its five sigma threshold, has a higher standard of "proof" than almost any other discipline. In the social sciences, for example, two sigma is considered credible and three sigma is considered something close to divine truth.

This is simply the wisdom of collective experience, although it has a solid basis that is less widely appreciated.

For example, conventional wisdom in high energy physics is that three sigma anomalies actually end up not being true about half the time, even though a three standard deviation from the mean anomaly in a Gaussian distribution should happen only about 1% of the time.

In contrast, it is exceedingly rare for a five sigma result to turn out to be a "fluke" (although in practice, informed by the wisdom of experience, for a result to be accepted it must not only meet the five sigma test, but must also be independently replicated and must have some sort of plausible theoretical justification, even if that justification was previously just a conjecture). The replication test guards against extreme cases of experiment specific unrecognized systemic error. The theoretical justification requirement guards against extreme coincidences which are more common that generally appreciated.

In statistical theory, three or four sigma ought to be plenty to assure that something is not due to random chance.

There are two main factors that make a lower threshold like three or four sigma insufficient in physics.

First, it is difficult to properly account for "look elsewhere effects". If you do twenty experiments with a Gaussian (i.e. bell shaped "normal") probability distribution, the odds are that one of the twenty will be more than two standard deviations from the mean, merely due to random chance. If you do enough experiments (and HEP experiments do a lot of experiments), a certain number of anomalies are statistically expected. But determining how many anomalies are expected as a matter of random chance is as a practical matter hard to quantify (since the definition of an "experiment" that should be counted is non-obvious) and is routinely underestimated.

In the language of social scientists, this higher threshold is designed to prevent p-hacking (in which p is the the likelihood that a result would be produced by random chance and the goal is to get p<0.05 which is the customary threshold for a statistically significant result). But still, generally speaking, social scientists do far fewer distinct experiments than physicists do, so the "look elsewhere effect" is less of a concern and the same high five sigma threshold is not required to overcome it.

Second, empirically extreme outcomes in physics experiments are actually not Gaussian (i.e. they don't appear in frequencies that match the bell curve of a "normal" distribution). Instead, they have "fat tails" (this is also true, for what it is worth, in election outcomes relative to political survey results).

This is mostly due to the fact that in physics some error is statistical (which is strictly Gaussian, but subject to the look elsewhere effects discussed above), and some error is systemic. There is no good reason that systemic error is or should be Gaussian, except that this is a convenient assumption to make, because it allows scientists to quantify systemic and statistical errors in ways that are easily combined mathematically.

The five sigma threshold prevents systemic errors that have "fat tail" probability distributions (like certain student t-test distributions) from being treated as "discoveries" that can be relied upon when they are actually less exceptional than a straight map of "sigmas" of error significance are mapped uncritically to a Gaussian distribution would suggest.

Fat tails are less of a concern in many social sciences because often variables that can have a range of values are calibrated to a Gaussian distribution because, while it is often possible to rank a quantity (like IQ) there is no well defined other way to quantify how much higher ranked one outcome is than another. Physics, with intrinsic means of quantifying things, in contrast, rarely calibrates its measurement system to be Gaussian artificially.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.