How to work out with how many sigma confidence my measurement confirms the theory?

Question

Suppose by a theory a certain value should be 1. I make an experiment in which I measure it to be 1.1 ± 0.2, where the ± 0.2 are 1 sigma Gaussian errors?

Obviously I can see that the theoretical value is well within 1 sigma of my measurement—at 1/2 sigma to be precise. So in this understanding the less sigma the better.

But if experimentalists make a statement such as "confirmed with n sigma", or "n sigma detection", then usually the larger n, the better.

What am I missing, and could you please help me understand at my simple Gaussian error example above, how to work out at how many sigma my measurement supports the theory. Cheers!

Answer 1

The example you have given is completely different from "confirmed with n-sigma".

In your example, theory predicts 1, and the experiment agrees, with a roughly 20% error bar. That's it.

Now if the experiment had measured $2\pm 0.2$, we could talk about ruling the theory out with a certain confidence level, which can be subtle and involved.

The idea of detecting a signal at $k$-sigma goes a follows: particle physics detectors don't see new particles, they see the decay products of new particles, and the decay products have the same signature as many well understood background processes.

So you make a histogram and look for a bump over background. For simplicity, consider a 1-bin bump. Your Monte Carlo model, which combines the Standard Model with your detector, tells you there will be $N$ background events in the bin, but your data has $T$ events (where $N$ and $T$ are integers, we are counting events).

If $T>N$, you have signal! You created a new particle $S$ times, where $S$ is:

$$ S = T - N$$

Suppose $N=100$ and $T=105$? Do you really believe those 5 events are your Nobel Prize? No, and neither does anybody else.

Both processes are subject to the usual rules of random counting. The expected statistical fluctuations are:

$$ \sigma_N = \sqrt N $$ $$ \sigma_T = \sqrt T $$

The uncertainty on $S$ is then:

$$ \sigma_S^2 = \sigma_T^2 + \sigma_N^2 = T + N $$

You're measurement of the signal is then:

$$ S \pm \sigma_S = [T-N] \pm \sqrt{T + N} $$

and the "number of sigmas" with which you have discover the new particle is:

$$ \frac S {\sigma_S} = \frac{T-N}{\sqrt{T +N}}$$

If we look at the Higgs Boson discovery (and sum over bins to reproduce my "1 bin histogram" analysis):

we can eye-ball (I am sure they were more careful before the announcement), we can eye-ball:

$$ S \approx 240 $$ $$ N \approx 3000$$

so that

$$ \frac S {\sigma_S} = \frac{3240-3000}{\sqrt{3240 + 3000}} =\frac{240}{6240} \approx 3$$

(Note: the real data has background constrained to better than $\sqrt N$...it's not a "1 bin" analysis).

There are several take aways here. Your SNR falls as the square root of your runtime, so a 3rd year of data would have been only a 20% improvement. It also pays huge dividends to find ways to reject background, esp. when your signal-to-noise ration is 1/12 (1 Higgs for every 12 background events).

Answer 2

In particle physics the signal of a "new particle" is usually covered by lots of "known effects" gathered from many sensors. Each sensor/detector has an uncertainty, and the "known effects" are only known up to a certain level. Therefore, after removing all known effects, the remainder of the signal might be due to chance. Statements like "we discovered the Higgs boson with an 5 sigma confidence level" should be read as "assuming that there exists no new particle, and that the signal is due to chance, we observed an extremely rare event -- $5\sigma$ correspond to a probability of $0.57ppm$. Since this probability is so low, we believe that our hypothesis (=there exists no new particle) is wrong.".

In most other fields in physics (particle and astro physics are probably the exceptions) the data analysis differs significantly from the one described above. In most fields the existence of an effect is rather obvious, and therefore, must not be argued/ backed up with a sigma level. E.g., taking your first example: You are not concerned with the existence, but with the location and its spread. These are different kind of questions. Therefore, we usually present them using confidence levels -- as you did in your example.