Why is the standard uncertainty defined with a level of confidence of only 68%?

Question

The widely used and accepted form of representing the uncertainty of a measurement is described in the GUM (Guide to the Expression of Uncertainty in Measurement).

There it is recommended to use the standard deviation as the given uncertainty of a measurement. (see the NIST Summary) I.e. the true value is in the given interval with a level of confidence of 68%.

Ever since the introduction of the uncertainty in my undergraduate courses I wondered why we are using such a low confidence level. I often see plots of measurements where for many values the theoretical prediction does not lie within the error bars. Seeing this always suggests at first glance that either the measurement was flawed or the theory is flawed. But in reality it's just our way of defining the error interval that seems to be flawed.

In extreme cases 1/3 of the values with their respective uncertainties do not fit to the theoretical prediction. We could easily multiply the uncertainties with a factor of 2 or 3 to make our confidence value 95% or over 99%. (see NIST definition of the coverage factor)

In my opinion a confidence level of 99% would be much more useful in evaluating the quality of a measurement. And in plots almost every measured value would show a correlation to the theory.

Are there good reasons to use the coverage factor of 1 as the standard representation of uncertainties or is it only a matter of convention/tradition?

And even if nearly all physicists would agree to use a higher coverage factor, would it be wise to switch to new definitions after we used this definition in thousands of papers already?

Answer 1

We talk in terms of standard deviation because this is traditionally the quantity you use to specify the variance of a Gaußian distribution specifically and any random distribution more generally. You seem to be misinterpreting the recommendation that all uncertainties be reported as standard deviations as a guideline that this should also always be what determines our demand for the "right" confidence interval.

Reporting standard deviation does not mean that all fields of physics consider the confidence interval corresponding to one standard deviation as a "good fit" - for example, the commonly used "gold standard" for reliable discoveries in high-energy physics is "5 sigma", i.e. in practice people are doing exactly what you propose and multiply the standard deviation by a factor that enforces a much higher confidence level. But in other fields, 5 sigma might be unrealistic e.g. due to experimental limitations or more sources of uncertainty and they use weaker bounds instead.

Giving the standard deviation is simply the standard way to report uncertainty, but it does not imply that in any particular field this is the actual bound used to determine whether or not a result is reliable. Reporting uncertainty in this standardized way is valuable because it makes results comparable across time and space without having to worry about what definition of uncertainty a particular source is using, but the interpretation of that uncertainty is highly variable.

Answer 2

Are there good reasons to use the coverage factor of $\pm 1\sigma$ as the standard representation of uncertainties or is it only a matter of convention/tradition?

The reason is convention however with full respect to the statistically rigorous (correct) way that we should use the reported values in comparisons.

Consider first that, when we establish a convention that everyone is to report $\pm 1 \sigma$, we do so so that everyone can easily obtain any multiple range that we want at any point without the need to question the source of the value now or at any point in time later. Convention here establishes a ground rule because $\pm 1\sigma$ is the absolutely least ambiguous report that can be made, both in terms of understanding across any discipline as well as understanding from now until all future times.

Consider second that, when we make comparisons between two values, we are never wise to ask whether the two values agree. We are far better to ask how confident we are that those two values differ from each other. When we have the $\pm 1\sigma$ standard range for two values, we can compare them rigorously to state the percentage confidence that we have that those two values differ to the given degree. When we want to increase our confidence in the difference above 68%, we go to a higher range in $\pm \sigma$.

Here is a specific example. Suppose that theory expects a value as exactly 4 and we obtain $4.5 \pm 0.2$ from a finite number of measurements on a given system. Our measured value is different from theory to a confidence of 68%. In 100 truly random measurements on our system, we should expect to obtain 68 values that predict that our system has a different true-mean from theory. When we increase to $\pm 3\sigma$, we find that 997 out of 1000 measurements on our system support a conclusion that the true infinite-distribution mean measured in our system will be different than theory predicts. We can therefore state that we are 99.7% confident that our value is different from theoretical expectation.

The second lesson is also illustrated by the somewhat amusing weather story. You can be highly confident when you might tell someone that tomorrow's weather will not be significantly different than today's weather. But you would only be modestly assured when you might tell someone that tomorrow's weather will be the same as today's weather.

Answer 3

The standard deviation is just one measure of uncertainty. Sometimes results are summarized as a mean (best estimate) and a standard deviation. If all you have is a set of data, the mean and standard deviation of the data can be easily calculated without any assumptions as to the underlying probability distribution. This does not imply that the mean plus/minus the standard deviation has a very high probability of containing the random variable.

For applications requiring small uncertainty, probability distribution percentiles are frequently used. For a random variable $X$, the specific value $x_p$ such that $Probability(X \le x_p)$ is called the $100pth\enspace percentile$. For small uncertainty the $95^{th}$ of the $99^{th}$ percentile may be used.

Also, instead of the mean the median is sometimes used for a point estimate; the median is the $50^{th}$ percentile. For cases where the estimated mean from a sample is $0$ (no occurrences out of $N$ trials), the median is a better point estimate because in this case the mean is insensitive to the number of trials (mean is always $0$ with no occurrences in any number of trials).

Be careful in the use of "confidence". Confidence is not the same as a percentile, although it is sometimes used incorrectly to mean a percentile. "Percentile" is based on a probability distribution for a random variable. "Confidence" is associated with an interval for a parameter of a probability distribution (mean, percentile, etc.) calculated from a sample, to indicate uncertainty based on a finite sample. For example, I can calculate a confidence interval for the $95^{th}$ percentile based on a sample. In basic probability courses it is assumed we "know" the probability distribution; in reality we frequently do not know the distribution, but we may assume the type of distribution (normal, exponential, beta, etc.) based on physical insights and infer a confidence interval for the parameters of the distribution based on samples.

If you "know" the probability distribution you can speak of its mean and percentiles. If you infer parameters of a probability distribution from a sample you speak of the confidence in the inferred values, using classical statistical inference.

(A Bayesian approach treats probability itself as a random variable in contrast to classical statistical inference where the probability is treated as a specific but unknown value- the number of occurrences out an infinite number of trials.

Answer 4

If you study probability theory, you will rarely find references to the standard deviation $s$ . The reason is that the standard deviation is a biased estimator: On average the samples' standard deviation $s$ is smaller than the populations' standard deviation $\sigma$, see example below. Therefore, mathematicians usually prefer the sample variance $s^2$, which is an unbiased estimator of $\sigma^2$. However, the variance does not have the same unit as the average value $\bar x$:

• If we measure distances in meters the variance would have the unit $m^2$.
• In contrast the average and the standard deviation has the unit $m$.

Thus, using the standard deviation (instead of the variance) is very convenient, because it allows us to perform simple insanity checks without using a calculator. Also, there are methods to compensate the bias.

Two remarks:

1. While the $\pm1\hat\sigma$ value is often used, the document you linked states that you should explicitly state what you mean by an expression such as $1.3\pm 0.1$. Therefore, there is nothing which prevents you from stating that the $\pm 0.1$ correspond to the 95% confidence interval. In fact the $95\%$ confidence interval is the "standard" for plots and hypothesis tests.
2. Whenever the distribution is asymmetric about its mean value, the samples' standard deviation is not a very useful quantity. Often quantiles would provide a "better" descriptive statistics.

Recommendation: Decide for yourself which descriptive statistics are best for your dataset. Whenever you feel that a 95% CI is better suited, please go ahead and use it.

Simulation: I drew $n=5$ random numbers from the normal distribution with $\sigma = 1$ and calculated the samples' standard deviation. Then I repeated this $10\,000$ times and generated the following histogram. The average sample standard deviation $E[s]$ is smaller than $1$. Not much, but the deviation is statistically significant.

Answer 5

The 68% confidence level is chosen because it corresponds to 1$\sigma$ of a Gaussian distribution. Therefore, if the distribution is Gaussian, converting from the 68% confidence level to the 95% confidence level is done by simply multiplying by 2 (more on this). In metrology, distributions are often assumed to be Gaussian since the central limit theorem suggests that the means of measurement results will be Gaussian distributed.

The choice of always presenting the 1$\sigma$ uncertainty is convention. But, it is a convention that makes since. Converting to any other confidence interval is straight forward. Additionally, a if the data are in good agreement with the theory, then the uncertainty bars will overlap at the 68% level. If they do not, that implies it is either a borderline result or it actually does not agree.

Answer 6

In the case of a bell curve representing the entire population, or the mathematics of a phenomenon… — any applicable bell curve — the concept of a standard deviation is the point at which the curve changes from getting more and more steep to getting less and less steep.
In other words… starting at the middle, and moving steadily away from the centre… the likelihood decreases, and decreases faster and faster… to the “standard deviation” point… and from there the likelihood decreases now slower and slower. This point exists, at which the rate of change (in the likelihood) changes from positive to negative… as a meaningful, inherent concept.

This point marks a conceptual boundary. It is reasonable to think of this conceptual boundary as delineating typical and not typical.

This “standard deviation” can be treated as a handy unit, and points on a bell curve can be represented in this new unit. However, that is entirely eisegetical; only the original standard deviation point is conceptually “real”.

These “typical” cases happen to represent about 68% of all cases. This 68% is used for Standard Uncertainty because it is the only point or measure that is “real”. Anything within this range is “typical” and anything outside of this range is not. The idea of this “Standard Uncertainty” is that a reading should be typical [with all of the above meaning packed into this word “should”].

The concept of a “Standard Error” exists, which can be made smaller by doing repeated experiments and measurements (which requires correcting for these being samples), but again there must be this root concept of the Standard Error of one sample or population. Using, for instance, 99% for the basic error of one measurement for the Standard Uncertainty would be very wrong.