Combining multiple theories with 5 $\sigma$ confidence level

Sadly I am not a physicist but I am interested in the topic. Please have mercy with me if you find my question trivial or dumb. Here it comes:

As far I understand physicist express their certainty about something with multiples of "sigma". The Higgs Boson exists with a confidence level of 5 sigma. I tried to understand what 5 $\sigma$ means. Here is what I think it means:

If you have a theory and you try to prove it you need an experiment. You then repeat the experiment over and over again and measure what you see. If your measurements tell you that what you observed is what you expected with a probability of 99.9999426697% (5 $\sigma$) then you can declare your theory a correct and you have a discovery. It that correct so far?

In science you are almost always using the work of others. What if your 5 sigma theory depends on a lot of other 5 $\sigma$ theories? Won't this reduce the probability of correctness? Let's say the "Higgs Boson Theory" depends on 10000 other 5 $\sigma$ theories. Wouldn't that reduce the probability of correctness by a factor of 10000? Isn't that a problem? Wouldn't it be better to vary the needed confidence level depending on how many different theories/discoveries your own theory depends on?

-

If you have a theory and you try to prove it you need an experiment. You then repeat the experiment over and over again and measure what you see. If your measurements tell you that what you observed is what you expected with a probability of 99.9999426697% (5 σ) then you can declare your theory a correct and you have a discovery. It that correct so far?

Almost. I'd put it more like this: suppose you have a theory, and you want to do an experiment to prove it. You would actually calculate what you expect to see if your theory is wrong. And not just a number, either, but a whole probability distribution that tells you how likely any given result is if the theory is wrong.

Let's take the Higgs search as an example. In this case, the theory is "a particle with a mass of $125\ \mathrm{GeV}/c^2$ exists." So the scientists at ATLAS and CMS would calculate something like the following (I'm just making the numbers up):

• If there is no particle with a mass of $125\ \mathrm{GeV}/c^2$, we expect to see, on average, 511 photon pairs of energy $125\ \mathrm{GeV}$ over 3 months of running the LHC. This is the "expected value."
• If there is no particle with a mass of $125\ \mathrm{GeV}/c^2$, we have a 68% probability of seeing between 506 and 515 photon pairs of energy $125\ \mathrm{GeV}$ over 3 months. This is the $1\sigma$ confidence interval.
• If there is no particle with a mass of $125\ \mathrm{GeV}/c^2$, we have a 95% probability of seeing between 501 and 521 photon pairs of energy $125\ \mathrm{GeV}$ over 3 months. This is the $2\sigma$ confidence interval.
• (skip a couple)
• If there is no particle with a mass of $125\ \mathrm{GeV}/c^2$, we have a 99.9999427% probability of seeing between 487 and 533 photon pairs of energy $125\ \mathrm{GeV}$ over 3 months. This is the $5\sigma$ confidence interval.

Of course, they do this with formulas, rather than writing out a list.

Anyway, then they run the LHC for 3 months and see how many photon pairs at $125\ \mathrm{GeV}$ show up. Let's say they got 534. The probability that this many photon pairs would be produced without there being a particle of that mass is pretty miniscule. That's how they're justified in concluding that there is a particle with mass $125\ \mathrm{GeV}/c^2$.

In science you are almost always using the work of others. What if your 5 sigma theory depends on a lot of other 5 σ theories? Won't this reduce the probability of correctness? Let's say the "Higgs Boson Theory" depends on 10000 other 5 σ theories. Wouldn't that reduce the probability of correctness by a factor of 10000? Isn't that a problem? Wouldn't it be better to vary the needed confidence level depending on how many different theories/discoveries your own theory depends on?

Actually, they do. The probability that the experimental confirmation of the existing theory is wrong is incorporated in those ranges I mentioned earlier. But you should also remember that, by the time they got around to testing the existence of the Higgs boson at the $5\sigma$ level, the other results that support the standard model has been measured way more precisely. So the standard model itself (other than the Higgs) is no longer confirmed at just $5\sigma$; it's confirmed at, say, $20\sigma$. The probability of it being wrong is entirely negligible.

-
Is it possible to have any realistic values for the first or second part of the answer? – deadly Jan 17 '14 at 11:50

This is not what sigma means, sigma means the opposite. It's a measure of how likely it is that the known background stuff produced the signal that you see by random chance.

1-sigma means that the probability that the event is due to random chance is 33%, 2 sigma means 5%, 3 sigma means .3%, 4 sigma means .006% and 5 sigma means .00005%, so less than 1 in a million chance that it was a fluke.

To understand this, suppose you are looking at a blue chair in a white room, but the room is dim, so you only see a few photons. The white floor sends out all colors of photons, while the chair only sends out blue. The chance that the floor sends out a blue photon is 10%. You see a blue photon, then another, then another, all coming from a spot in the room. You then see a red photon coming from another spot and a blue photon from another spot.

You reach 5 sigma confidence when you see about 10 blue photons from the same chair-shaped region in the room, and no red photons. Supposing it is categorically impossible for the chair to emit non-blue photons, you reach 5 sigma certainty that a chair does not occupy a spot when you see one non-blue photon coming from that spot (actually, infinite sigma certainty).

Why do I say you need 10 blue photons and not just 6? Six photons is 1 in a million after all, isn't it? It's because of selection bias--- there might be 10,000 different qualitative places where the chair can be, or there might be no chair, and you are selecting through the photons, looking for clumps of blue. If you look at 10,000 clumps of 4 photons, you will find 4 blues just by random chance. So these 4 photons don't count. The next six give you the 5 sigma certainty.

The reason scientists set the bar for a discovery at 5 sigma is because experimental groups sift through mountains of data, and they are looking for interesting deviations. Just by random chance, like finding 4 blue photons from a place where there is no chair, they find things that are superficially 2 or 3 sigma significant all the time, and these things are nothing at all. These things are not really 3 sigma significant if the probability were accounted for correctly, because of all the searching and sifting that went on to find them.

If experimentalists could easily quantify human tendency to find patterns, we could probably get by with a "real honest-to-goodness 3-sigma" confidence level. But given that there are so many possible patterns: you didn't see the blue photons but you saw 5 red photons--- maybe the chair is painted red! You saw red and yellow photons--- maybe the paint is orange! You have a lot of theory freedom. So you easily can get 3 sigma evidence for some theory you make up to fit the data. So if this is your theory, you need to wait for 5 more red photons before declaring the result, just to be sure you really see a red chair.

The way it works in practice is that you need to make a prediction based on 3 sigma data that is then confirmed by independent 3 sigma data, and then you are confident. But it always helps to replicate the result. The 5 sigma threshhold is pretty safe--- people have been burned by 4 sigma and 3 sigma events many times, but not so much by 5 sigma.

This sort of thing is very useful for the general public. If there is a catastrophic isolated event, like a financial meltdown, a terrorist act, or a string of crimes, it is good to get a sense of what the sigma is on the event before changing policy drastically. It could have been an unlucky coincidence.

-