How to combine measurement error with statistic error

Question

We have to measure a period of an oscillation. We are to take the time it takes for 50 oscillations multiple times.

I know that I will have a $\Delta t = 0.1 \, \mathrm s$ because of my reaction time. If I now measure, say 40, 41 and 39 seconds in three runs, I will also have standard deviation of 1.

What is the total error then? Do I add them up, like so?

$$\sqrt{1^2 + 0.1^2}$$

Or is it just the 1 and I discard the (systematic?) error of my reaction time?

I wonder if I measure a huge number of times, the standard deviation should become tiny compared to my reaction time. Is the lower bound 0 or is it my reaction time with 0.1?

Answer 1

Yes, the only sensible formula for the total error is the sum in quadrature, $$ \Delta X_{\rm total} = \sqrt { \Delta X_{\rm syst}^2 + \Delta X_{\rm stat}^2 } $$ The key assumption behind the validity of the formula is that the two sources of error are independent i.e. uncorrelated. $$ \langle \Delta X_{\rm syst} \Delta X_{\rm stat} \rangle = 0$$ Because of that, we have $$\langle \Delta X_{\rm total}^2 \rangle = \langle (\Delta X_{\rm syst} +\Delta X_{\rm stat} )^2 \rangle = \sigma_{\rm stat}^2 + \sigma_{\rm syst}^2$$ The term $2ab$ from $(a+b)^2=a^2+2ab+b^2$ drops out because of the independence quoted in the previous displayed equation. The last displayed equation is a full proof of your formula.

I want to emphasize that the Pythagorean formula doesn't depend on any normality of the distributions. It's just simple linear algebra used in computing the expectation value of a bilinear expression in which the mixed terms contribute zero because of the independence above. If someone tells you that you have to assume the central limit theorem or Gaussianity of the distribution, she is just wrong.

Of course, if one wants to convert the information about the error margin to $p$-values, i.e. confidence levels, one needs to know the shape of the distribution, i.e. to assume it is Gaussian. If both the systematic and statistical error are distributted via the Gaussian distribution, so is the total error. But if we don't talk about $p$-values, we don't need to assume anything whatsoever about the Gaussianity.

However, it's very useful to separate the systematic and statistical error because if you repeat some measurement with the same equipment, the statistical error adds in quadrature but the systematic error adds linearly.

This statement means that the statistical errors from independent "runs" of the same experiment are uncorrelated with each other $$ \langle \Delta X_{\rm stat1} \Delta X_{\rm stat2} \rangle = 0$$ and they're still uncorrelated with all the systematic error, too. However, the systematic errors are linked to the device which is still the same, so the systematic errors from 2 repeated "runs" are perfectly correlated: $$ \langle \Delta X_{\rm syst1} \Delta X_{\rm syst2} \rangle = \sigma(\Delta X_{\rm syst1})\sigma(\Delta X_{\rm syst2}) \neq 0$$ On a $2D$ plane, the distribution function would be concentrated near the "diagonal" tilted line $\Delta X_{\rm syst1} = k \Delta X_{\rm syst2} $. Be careful, under some conditions, the result above would need a minus sign. This linearity makes a difference. In particular, the statistical errors for "intensive quantities" may be reduced by repeating the experiment while the systematic errors can't.

Imagine that the LHC measures the decay rate of a particle as $\Gamma=CP$ where $C$ is a fixed constant without error and $P$ is the percentage of their events (collisions) that have a certain property. Let's make two runs with $n_1$ and $n_2$ events, respectively. They're expected to give the same number $n$ and the total number is $N=2n$.

However, the first run has $\Delta n_1$ with both statistical and systematic component and the same for $\Delta n_2$. What's the total number of collisions? We measured $n_1+n_2$ collisions but this result has an error margin (more precisely, I will be talking about the error margin of $\Gamma$ with the right coefficient). For the error we have $$ \Delta N = \Delta n_1+\Delta n_2 = \Delta n_{\rm 1stat}+\Delta n_{\rm 1syst}+\Delta n_{\rm 2stat}+\Delta n_{\rm 2syst}$$ What is the expectation value of its square? $$ \langle (\Delta N)^2\rangle = (\Delta n_{\rm 1syst}+\Delta n_{\rm 2syst})^2 + (\Delta n_{\rm 1stat})^2 + (\Delta n_{\rm 2stat})^2$$ Note that the statistical errors from the two runs were first squared and then added; for the systematic errors, they were first added and then squared. As the result, the systematic contribution to the error of the decay rate won't change when you make another, second run. The statistical error will drop by the factor of $1/\sqrt{2}$. Because the different parts of the total error behave differently, it's good to know the errors separately.

But if you only use an apparatus or setup once and then you destroy it, there's no reason to remember the separation and the right total error margin is obtained by adding them in quadrature. That's what many high-energy experimental teams did and the reason is not that they're sloppy about statistics. The Pythagorean calculation is perfectly valid and may be used by those who know what they're doing. Just the "beginners in statistics" at school are discouraged to combine these things in quadrature because they could add the errors incorrectly if they consider many measurements with the same device.

But adding the systematic and statistical error margins linearly would always be wrong because they're always independent of one another. It would produce a larger numerical value of the error margin than the Pythagorean formula and a larger error is found "OK" by some people because it makes the experimenters sound more cautious or "more conservative". But it's still a wrong result, anyway. If someone found a 5-sigma evidence/proof for an effect using the Pythagorean formula for the error margin and you would deny her 5-sigma evidence/proof because you would calculate your error margin that is overstated (probably by the simple sum of the syst. and stat. error margins), therefore getting just 3 sigma, then you would be a denier of a valid experimental proof of an effect which is bad whether or not you can also claim to be "conservative" or "cautious". ;-)

There's only one right formula in science and for a single statistical and single systematic error, it's given by your Pythagorean formula.

Answer 2

I think you're exercising an incorrect picture of statistics here - mixing the inputs and outputs. You are recording the result of a measurement, and the spread of these measurement values (we'll say they're normally distributed) is theoretically a consequence of all of the variation from all different sources.

That is, every time you do it, the length of the string might be a little different, the air temperature might be a little different. Of course, all of these are fairly small and I'm just listing them for the sake of argument. The point is that the ultimate standard deviation of the measured value $\sigma$ should be the result of all individual sources (we will index by $i$), under the assumption that all sources of variation are also normally distributed.

$$\sigma^2 = \sum_i^N{\sigma_i^2}$$

When we account for individual sources of variation in an experiment, we exercise some model that formalizes our expectation about the consistency of the experiment. Your particular model is that the length of the string (for instance) changes very little trial after trial compared to the error introduced by your stopwatch timing. Unless we introduce other errors, this is claiming $N=1$, and if the standard deviation of your reaction timing contributes $0.1 s$ to the standard deviation of the measurement, then theoretically the measurement should have that standard deviation as well.

If this conflicts with the statistics of the time you actually recorded, then the possible ways to account for this include:

• Your reaction time isn't as good as you thought it was
• There are other sources of experimental error

I would favor the latter, although it could be a combination of both of them.

Answer 3

Errors are given so as to assign a probability of deviation from the true value, they are the uncertainty in measurement. There is some confusion about systematic errors, i.e. errors that do not come from statistical fluctuations but from the method of measurement used, "your measurement error".

Systematic errors are the dominant ones when the statistical become very small, as will be the case if you make very many measurements and your reaction time is left as the main uncertainty.

The best way in my opinion is to quote both errors and not to combine them in quadrature. If the final value is the result of a fit, as was the case for the mass and width of the Z boson for example, then the systematics are taken care during the fit and a single error is quoted.

At present, people tend to add all errors in quadrature, quoting a Central limit theorem which essentially argues that systematic errors also come from random normal distributions . One should point out that this is not always the case, particularly for scale errors as was amply demonstrated by the superluminal neutrino premature announcement. If they had included to their errors a +/-70 nanosecond systematic ( from a bad connection), they would not have thought that anything exceptional had been measured.