# What is meant by precision of a measuring instrument?

I came across many definitions of the precision of a measuring instrument such as, "a precise measuring tool is one that can measure values in very small increments" and "The precision of the instrument that is used to take a measurement is reflected in the smallest measure that the instrument can make". However, none of them provided precise information about what exactly is the precision of a measuring instrument?

• Also note that introductory textbooks usually define "accuracy" and "precision". But error analysis in the real world may use different terms in different fields. For example, in engineering it was common to use the concepts of "systematic error" & "random error". "Systematic error" is a predictable offset between the measured value and the true value which you can remove by calibration, while "random error" is unpredictable and cannot be calibrated. Meanwhile, BIPM's Guide to the Expression of Uncertainty in Measurement (GUM) uses a different system to define "Type 1/2" uncertainty. Jan 4 at 13:07

Let me tell you one more way to interpret precision: it's a measure of an instrument's capability to reproduce the same readings during trials.

The term precision is used to describe the degree of freedom of a measurement system from random errors. Thus, a high precision measurement instrument will give only a small spread of readings if repeated readings are taken of the same quantity.(source)

There can be many ways to say the same thing but its essence remains unchanged. Here, the essence of the word precision is the ability to be precise. Precise comes from Latin "praecis-" ‘cut short’, it's how sharply an instrument can measure something. If a tool lacks precision, readings will vary with each trial, whereas a precise tool will generate identical readings most times.

A precise instrument may or may not give correct readings. The ability to give correct measurements is called "accuracy". An accurate instrument may not be precise. Accuracy is often confused with precision; here is a good illustration:

• The classic image. Jan 4 at 1:47
• How is the image 3 ( Row 1 column 1) accurate ? Jan 4 at 12:08
• @An_Elephant The average is in the middle. In particular, the common practice of averaging noisy measurement results to get a better estimate works only if the measurement device is accurate.
– JiK
Jan 4 at 13:02
• I think that one thing worth adding is that while precision is basically standalone (i.e. the diameter of a group of shots on a featureless board) accuracy relies on some external standard (i.e. that the board has a target pinned to it). Jan 4 at 13:06
• To piggyback off of Mark's comment, note that if you zoomed in on the "yes / yes" image, you would get something that looks a lot like the "not precise, but yes accurate" image. It's not really about a yes/no answer, it's more about how much it's precise. For example, one doesn't generally say "this ruler is precise" in an absolute sense, you say "this ruler has a precision of 1mm" (or "this ruler is precise to within 1mm"). Jan 4 at 18:24

I think quote

The precision of the instrument that is used to take a measurement is reflected in the smallest measure that the instrument can make.

says it all. Say you have such ruler (one of instruments) :

Then smallest length it can measure is $$1~mm$$ (omitting zero of course). Because ruler has no smaller scale ticks marked anymore, than 1 mm, hence this instrument system error is $$\pm 1~mm$$. When we measure length with this type of ruler we say that our obtained object length was $$L \pm 1~mm$$.

Sometimes system error is considered to be half of smallest measuring step. Consider that real object length is $$0.5~mm$$, however this ruler can't measure such small object, and so you are "forced" to choose $$0 mm$$ or $$1mm$$ value for an object length by applying this ruler measurement. And so you are "off" by $$0.5mm$$ from a real value. Hence you can state that your object length measurement error was $$0.5mm$$.

This is the rationale why sometimes system error of measuring instrument is considered $$\frac 12 \Delta_{min}$$ instead of simply device $$\Delta_{min}$$.

• I'd say that's more the resolution rather than precision. Granted, excessive resolution without precision makes all the extra resolution useless. In instruments of reasonable quality, precision will be on par with the resolution, but I shouldn't need to say that not all instruments are reasonable. Jan 4 at 2:04
• ...why does that ruler have the 2 times table printed on it? That feels like the one people would least need as a reference, save the 1 times table. Jan 4 at 2:40
• @DKNguyen Good note. Yes, resolution greater than precision totally makes no sense, so I was thinking that all manufacturers knows and respects this, but maybe I simply have not seen "crazy instruments" much :-D Jan 4 at 11:26
• @DKNguyen To prove that having excessive resolution greater than systematic precision is not only useless, but actually silly. Imagine, that this ruler is able to only measure within $1~cm$ precision, but still has $mm$ grades on it. So when I will "measure" with it say $2mm$ object length, I shall say that length of object is $2mm \pm 1 cm$. Isn't that stupid ? Measuring instruments manufacturers should never let resolution to be greater than systematic error allows, for this reason. Jan 4 at 14:12
• This is how it was explained to me when I was a physics major in undergrad. I still think it's a really relatable and intuitive way to explain the basic principle. My professor noted that we could estimate one order of magnitude below the markings as well. This was very introductory, and professional standards are likely higher. Jan 4 at 22:17

Thanks to the OP for the interesting question related to measuring physical quantities and metrics related to those measurements. Data sheets for sensors provide specifications based on metrics which can be diverse in definition. Understanding the relationship between the metrics is useful in experimental physics or engineering, and can help ascertain the intuition motivating the concepts of such metrics.

We present a brief but insightful comparative understanding of the most commonly used metrics to understand the limitations of sensors commonly studied by application of statistical metrics. Resolution, precision and accuracy are types of metrics or indicative classification classes of such metrics. A resolution metric expresses the smallest discernible difference or increment between two values that can be distinguished or resolved by a measurement or a system. An accuracy metric expresses how close a measured or observed value is to the true or actual value of a quantity. Usually, for any sensor, for a specific metric definition, the absolute resolution is lower than the absolute accuracy (error) or $$\text{absolute resolution} < \text{absolute accuracy (error)} \\ \equiv \text{percentage absolute resolution} < \text{percentage absolute accuracy (error)},$$ wherein the percentage absolute accuracy (error) or percentage absolute resolution refers to the ratio of the absolute values with the range of the sensor. Simply expressed, this means that we cannot be more accurate than the discrete quantity which we can measure. Indeed, in case of measurement of physical quantities by sensors this relationship is always true and it is difficult to come up with an example where this is not true.

In the context of sensors and measurements, precision refers to the degree of consistency or repeatability in a sensor's measurements when the same input or quantity is measured multiple times under similar conditions. Let's say we have a temperature sensor that, when exposed to the same temperature repeatedly, provides very similar output values each time. This indicates high precision. For instance, if the sensor measures a temperature of $$25^o \; C$$ and consistently outputs values like $$250.1 \; mV, 250.2 \; mV$$, and $$250.0 \; mV$$ over multiple measurements, it demonstrates high precision while if the sensor's output values for the same temperature vary widely such as $$245.0 \; mV, 253.5 \; mV, 246.8 \; mV$$ it suggests lower precision. Precision focuses on quantifying the consistency and reproducibility of measurements. A sensor that is precise produces similar measurements for the same input, indicating that it has low variability in its readings under similar conditions.

Overall, the absolute precision (error) is unrelated to the true measurement and is best described by the mean absolute deviation (MAD) or variance or standard deviation (STD) of the measurements while the absolute accuracy (error) is described by the mean absolute error (MAE) or root mean square error (RMSE) of the measurement relative to the true measurement. In practice, a sensor can be

• precise but not accurate (consistent but not necessarily correct),
• accurate but not precise (correct on average but with significant variability),
• both accurate and precise (consistent and correct),
• or neither accurate nor precise (inconsistent and incorrect).

The ideal situation is to have both accuracy and precision, ensuring measurements are both correct and consistent. The schematic below shows the distinction between these two metrics in the context of sensor measurements.