I am aware of the 'one sigma', 'two sigma' etc 'level of significance'. However this description of the significance of a result with sigmas seems to say 'If this other theory was not correct, then there would be only an $x\%$ chance of this experimental result'. For example, with the Higgs boson 'discovery', the result was significant to the 5 sigma level because there was only a one in 3.5 million chance of the Higgs boson-like 'hump' that was seen simply being part of the background noise.

Suppose you are trying to test how closely a value predicted by a theory matches experiment. You perform the experiment and get, say, 0.998$\pm$0.014 and the predicted value is 1. Is there anything that you can say about this particular data to judge how closely the experiment fits with the data (e.g. can you divide the error up and say that the experimental value is within a quarter of a standard error)? Or are you simply limited with this given data to saying that the experiment is consistent with the theoretical prediction within the error, and then to judge more closely how well the prediction matches with the real world you would need to conduct higher precision experiments or take more repeats to reduce the standard error in your experimental value?

  • $\begingroup$ A quoted error such as $\pm 0.014$ usually is taken to mean that the probability distriubtion around the quote value is Gaussian and has a half width half maximum of $0.014$. This is conventional, but of course how you want to do it depends manifestly on how you did your experiment. Besides: I don't understand the question. $\endgroup$ – Wolpertinger Apr 20 '17 at 15:15
  • $\begingroup$ Some experimental results are inherently statistical (you throw a dice and want to know the likelihood of getting a 1) you can get some result with some probability. You cannot predict with certainty the result of the experiment. What would be, to you, a good measure of precision in such scenario? $\endgroup$ – user126422 Apr 20 '17 at 15:22
  • $\begingroup$ @ArmandoEstebanQuito Thank you for your reply. I just thought that maybe you could say how close it was to the actual value within the error? For wxample 1.00$\pm $0.014 compared with 1.014$\pm $0.014. Technically both agree with the theory within their error, and I know nothing can be said conclusively about one providing stronger support than the other, but they certainly do not seem to be equivalent. $\endgroup$ – Meep Apr 20 '17 at 15:53
  • $\begingroup$ @21joanna12 again assuming Gaussian distribution of errors, one can say exactly how likely a given deviation from the theoretical prediction is, and then decide what the 'acceptable' cutoff is (which always must be an arbitrary decision, although there are conventions). This is standard stuff you can find in any error analysis textbook- I suggest that of Louis Lyons. In the general case things can get much more complicated than this. $\endgroup$ – Rococo Apr 20 '17 at 16:14
  • 2
    $\begingroup$ Part of the problem here is that you are not yet grappling with the complexity of the question. You haven't said what your landscape of hypothesis looks like. You haven't said anything about how hard it is to actually construct an apparatus that performs the experiment (how directly does it measure your quantity of interest, how well is it calibrated, how much does that calibration drift, ...). In the most general interpretation the questions is big enough for multiple books. $\endgroup$ – dmckee Apr 20 '17 at 16:46

As you've noticed, traditional physics analyses, including the discovery of the Higgs boson, use frequentist hypothesis testing. One calculates the probability of obtaining data at least as 'extreme' as that observed, were the null hypothesis (e.g., the standard model without a Higgs boson) true. This is known as a $p$-value. If the $p$-value is sufficiently small, we decide to reject the null hypothesis.

This says nothing directly about the plausibility of the alternative or null hypotheses, and, even if the null hypothesis is true, the $p$-value won't indicate that it's favoured by data. Even asymptomatically, it wouldn't converge, but make a random walk between zero and one.

There is, however, an alternative framework - Bayesian statistics - in which one may directly calculate the change in relative plausibility of models in light of data. This requires at least two competing explanations of data.

Suppose you measured a quantity, $D =0.998±0.014$. With Bayesian statistics, you can find the change in relative plausibility of two competing models, in light of that measurement. See e.g. my article about the notorious diphoton anomaly for a pedagogical example. The plausibility of a new massive resonance relative to the standard model increased by about 10 in light of early run-2 LHC data.

  • $\begingroup$ And even this structure assume the apparatus is well understood and calibrated and measures something that give reasonably direct access to the quantity that you care about. $\endgroup$ – dmckee Apr 20 '17 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.