Reporting machine resolution as experimental error

Question

I'm doing a measurement with a machine that has a resolution of $0.14 \mu\textrm{m}$. I can do said measurement many times and then calculate the average of measurements, which comes to $52.0094 \mu\textrm{m}$, with a standard deviation smaller than the resolution limit.

Can I report my value as $52.01\pm 0.14 \mu\textrm{m}$ or should I round the average to one significant value in the decimal position? Or would it be better to report the full uncertainty?

Answer 1

There are two simple situations:

1. If the standard deviation for the individual measurements ($SD$) is large compared to the resolution $res$, we can neglect the resolution. The distribution of the $SD$ determines the standard error of the average value $SE$. In the example below I assume a normal distribution and $N=10$ measurements, such that $SE = SD/\sqrt{N}$.
2. If $SD \ll res$, we can neglect $SD$. Assuming that every value within the length defined by the resolution has the same probability, we obtain a uniform distribution. Thus, the measurement uncertainty becomes $SE = res/\sqrt{12}$.

In between these extrem cases the exact location of the true value matters, and the interplay between the two components ($SD$ and $res$) becomes complicated. This can be seen by simulation.

In the code I define SD such that the ratio SD/res ranges from $10^{-4}$ to $10$, because I want to get equi-distant points in the log-log plot:

set.seed(17983)

nRepeat = 2e3       # repeated experiments
nSim    = 10        # number of measurements for a single experiment
res     = 0.14      # resolution measurement device
mu      = runif(nRepeat, min=20, max=80)  # true population value (random, but fixed)

# Now we want to change the standard deviation of the individual measurments
nSD = 100                                # number of SD we simulate
SD  = res * 10^seq(-4, 1, length=nSD)    # standard dev. of measurement 
SE  = vector(mode="numeric", length=nSD) # allocate space 
for ( i in 1:nSD ){
    err       = rnorm(nSim*nRepeat, mean=0, sd=SD[[i]]) # random error
    dim(err)  = c(nRepeat, nSim)         # reshape: each row is a measurement
    data      = mu + err                 # add true value to each row (dirty programming)
    data      = round(data/res)*res      # round to resolution
    E         = mean.rowwise(data)
    Delta     = E - mu
    SE[i]     = sd(Delta) # standard error
}

# Generate a log-log plot:
plot(log10(SD/res), log10(SE))
lines(log10(SD/res), log10(SD/sqrt(nSim)), col='red') # normal distr.
abline(h=log10(res/sqrt(12)), col='blue') # standard deviation for a uniform distr.

which yields the following

To obtain an intuition for the region $SD \approx res$ we take two perspectives:

1. Starting with the normal distribution and decreasing $SD$ the finite resolution increases the probability density in the edges. Effectively this increases the standard deviation of the normal distribution. That's why the dots are above the red line.
2. Starting with the uniform distribution and increasing $SD$ we "loose" some probability density in the edges of the uniform distribution, because the normal distribution shifts this density to the center. We can think of this situations as obtaining a uniform distr. with a smaller width $res$. That's why the dots are below the blue line.