Computing a quality factor of multiple measurements Suppose I measure the same quantity twice with two methods, first I get 0 with 0.001 uncertainty, then I get 1 with 0.000001 uncertainty. We can see from this most likely there is something wrong with the uncertainties or measurements. I am faced with such a problem, I have multiple measurements of the same quantity with prescribed uncertainties, and I would like to calculate how compatible these measurements are, or if they are correct, how unlikely it is to get those values. A quality factor or something is what I'm looking for.
What is the most sensible way to do this?
The goal is to know if there is something wrong with the uncertainties or not.
 A: If you assume the uncertainties are Gaussian distributed, then 1 is 1000 sigma away from 0 +/- 0.001. To a first approximation, you can ignore the much smaller uncertainty on 1.
You can use the complementary error function to determine the probability of finding a value at $n$ sigma away from the mean value (for Gaussian uncertainties).
For example,
\begin{equation}
{\rm 1 \sigma:} \ \ {\rm erfc}\left(\frac{1}{\sqrt{2}}\right) = 0.3173105...\\
{\rm 2 \sigma:} \ \ {\rm erfc}\left(\frac{2}{\sqrt{2}}\right) = 0.0455003...
\end{equation}
By the way, the $2\sigma$ result is very close to the $p$-value of 5% used in some sciences as a threshold for statistical significance.
In your example, we want to know the probability of getting a result at 1000$\sigma$ or higher. I evaluated this on WolframAlpha and got
\begin{equation}
{\rm 1000 \sigma:} \ \ {\rm erfc}\left(\frac{1000}{\sqrt{2}}\right) = 4.58 \times 10^{-217151}\\
\end{equation}
Suffice it to say, it is very unlikely your two measurements differ only by a statistical fluctuation. You should be looking for a systematic error affecting one or both measurements.
A: Measurements that use the same method can be assumed to have the same systematic error, affecting all measurements equally. So you can, for example, take an average value of these measurements or discuss the consistency of different measurements without knowing the systematic error.
However, it is not possible to compare measurements made with different methods and hence different systematic errors without knowing what these systematic errors are. For example, the two measurements in your example could be consistent with each other if one has a systematic error of $-0.5$ and the other has a systematic error of $+0.5$.
