3
$\begingroup$

I'm tasked with fitting a set of data to a line using Excel's $\text{LINEST}$ least-squares fitting routine. The data are $l$ and $m$ values, assumed to follow the relationship:

$$l = l_0 + {g\over k}m$$

...where $g$, $k$, and $l_0$ are some constants

Using $\text{LINEST}$, I obtain estimated values for $g/k$ (the slope) and $l_0$ (the $y$-intercept), reported to some absurd number of significant figures, in addition to uncertainties in both of those values as well as the $m$ values.

Let's suppose that $l_0$ is found to be $3.6857142... \pm 0.222953...$, $g/k$ to be $0.00607143... \pm 0.003742...$, and the uncertainty in the $m$ values to be $\pm 0.24250267...$, where the $m$ values are reported to $1$ significant figure (e.g. $500$).

To how many significant figures to I round each of these values?


Across undergraduate physics courses, I've been told many contradictory things about rounding and to how many significant figures one should report a measurement.

As I understand it, one would round the uncertainty in $l_0$ to just one significant figure (i.e. to $0.2$), then round $l_0$ to the same decimal place as this uncertainty (i.e. to $3.7$), and likewise for $g/k$. I'm not sure about the uncertainty in $m$, however. Should it be rounded to one significant figure, as well?

I've also been told, however, that the reverse is true (i.e. that one should first round the measurement/estimated value to a single significant figure, then round the uncertainty accordingly).

Which, if any, is most appropriate? Moreover, why?

$\endgroup$
3
  • $\begingroup$ What do the uncertainty in the m values reported in the output mean? I thought m is the independent variable. $\endgroup$
    – dllahr
    Feb 21, 2022 at 3:41
  • $\begingroup$ Example for $l_0$,- round to a first uncertain decimal digit, i.e. digit for which exists error digit greater than zero. So $l_0 = 3.7 \pm 0.2$ $\endgroup$ Feb 21, 2022 at 9:24
  • $\begingroup$ More on significant figures. $\endgroup$
    – Qmechanic
    Oct 15, 2022 at 17:53

2 Answers 2

4
$\begingroup$

The most authoritative source for how to estimate and report measurement uncertainty is the BIPM’s publication: Evaluation of measurement data — Guide to the expression of uncertainty in measurement. In section 7.2.6 it says

The numerical values of the estimate y and its standard uncertainty uc(y) or expanded uncertainty U should not be given with an excessive number of digits. It usually suffices to quote uc(y) and U [as well as the standard uncertainties u(xi) of the input estimates xi] to at most two significant digits, although in some cases it may be necessary to retain additional digits to avoid round-off errors in subsequent calculations.

(emphasis added)

So the recommendation is to typically use two digits to report your uncertainty. Then, it doesn’t make any sense to report your measurement to any higher precision. Thus your $l_0$ would be reported as 3.69(22)

A single digit for the uncertainty would be a little too imprecise. If you reported it as 3.7(2) then you would have to change your precision by 50% to have it change your report. That is too coarse.

Anyway you definitely decide on the number of digits to report for your error first, then that determines how many digits to keep for the measurement.

$\endgroup$
2
$\begingroup$

If you truly wanted to be rigorous, you could take the original $(x,y)$ variables (representing your data points), and using the regression formulas for $\alpha$ and $\beta$ here https://en.m.wikipedia.org/wiki/Ordinary_least_squares#Simple_linear_regression_model

compute the uncertainties via the formulas here https://en.m.wikipedia.org/wiki/Propagation_of_uncertainty#Example_formulae

Then you can use the uncertainties of the original data points to find the uncertainties in your slope and intercept.

The algebra and calculus is a little tedious, but that will give you the truest answer. And it will be useful to compare that answer to the recommended practice of 2 sig figs cited in the other answer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.