Error estimation during measurements with high standard deviation

Question

I want to measure average distance between fixed metal construction and water, as shown in the picture below to predict water flood. Let's call this distance water level h. If the water level starts rising, then I need to inform the local people that the flood is coming and they have to do something, etc.

By black color, I show fixed metal construction, which is not moving. The blue color is water under this metal construction. Let say water is a lake which always has some waves and never stays calm. and waves are not right Sin shape, but random.

I have an ultrasound/laser or any other measurement device which can measure the distance between the device and the water with 0.1cm error very fast (much faster than water waves changes, for example in 1ms). I do many measurements (100-200 times), and I calculate an average water level in relation to my metal construction.

For example, I got average h=123.2 cm after 100 measurements, but because water is always moving the standard deviation is high, like 20 cm.

In this example, can I tell that the water level h=123.2±0.1 cm or I can tell only h=120±20cm because the standard deviation is 20cm?

In other words, if today I get average h=123.2cm, tomorrow I will get h=130.5cm and the standard deviation is the same 20cm, then should I inform people that flood is coming or I cannot because water level difference is less than the standard deviation, that means it is below my error and I cannot really tell if water level is going up or down.

This is just an example to demonstrate the question. There is no real task like this. It can be replaced by another example (measuring cylinder diameter when it is not ideal cylinder) or something else where the error of the device is much less than the standard deviation.

Answer 1

Generally such problems don't readily work by a simple application of simple statistics. A standard deviation may not be particularly useful as an indicator. For example, during flooding the wave action may be very different to during steadier conditions.

You also need to know the generic nature of the flood process. Inflow into the lake increases the level all over the lake. Wind pushing the water to one side is very different but can still flood a portion of the lake shore. A water skier coming extra close to the dock may send a 1 meter wave across the dock, which probably should not set off your flood warning system.

You need at least a minimal model of total water in the lake as estimated by level measurements. Probably you need several level measurements at different locations. You need to have these over time in order to get the rate of change of water in the lake.

Then you need to figure out some way to deal with noise. Standard deviation may be useful but may not be. There are lots of trend measurements. For example, there are moving averages.

https://en.wikipedia.org/wiki/Moving_average

That page also gives links to a bunch of other possibilities.

Once you have a model of the total water in the lake you need test data to validate it. You would need to get real observations and compare them to when there was flooding. If your model is accurate time for some celebration. If your model is not accurate, back to work.

Answer 2

Assuming a normal distribution the chance of a new sample s being $n\sigma$ outside the mean $p_{outside}(s)$ is fixed.

You can see how that's used in the table here https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule

Consequently, before declaring a flood, select a value of $n$ that gives you sufficient confidence.

A sample with a $1\sigma$ deviation is 32% likely just to be due to a error (a big wave.)

It's popular to work to about

$3\sigma$ (.027% or likely to occur naturally every 370 samples)

but important results are usually confirmed to

$6\sigma$ (.000000002% or likely to occur naturally every 500,000,000 samples).

or higher.

Eliminating measurement errors will help to achieve a narrower distribution, improving confidence.

Answer 3

For example, I got average h=123.2 cm after 100 measurements, but because water is always moving the standard deviation is high, like 20 cm. In this example, can I tell that the water level h=123.2±0.1 cm or I can tell only h=120±20cm because the standard deviation is 20cm?

This is a case where actually looking at the data makes it clearer what's happening. Here's some data which has the characteristics you give: a mean of 123.2 cm, and a standard deviation of $\sigma = 20\rm\,cm$. I've assumed a normal distribution, but you can choose a different distribution if you feel like it. These thousand are plotted versus the measurement number:

The dashed lines are at zero, $\pm1\sigma$, $\pm2\sigma$, and $\pm3\sigma$ from the mean. You can see that most of the data lie within the $\pm1\sigma$ band around the mean, and nearly all the data lie within $\pm2\sigma$. Only very rare points lie outside of the $\pm3\sigma$ band. There happen to be exactly three measurements outside of the $\pm3\sigma$ band (near the middle, and all on the side approaching 200 cm), which someone who is new to this business might take as confirmation of the statement in another answer that 99.7% of normally-distributed data points lie within $\pm3\sigma$ of the mean. But the fact that I got exactly three "outliers," and that all of the outliers happen to be on the high side, is a fluke: three three-sigma outliers per thousand points is the average over many thousands of data points, and any particular thousand data points might have a few more or less than three outliers.

If I collapse these data into a histogram, it looks like this:

You can see here that a measurement of 130 cm is not uncommon at all; this data set has fifty or sixty measurements in the bin where a measurement of 130 cm would go. When you tell me $(123.2\pm20)\rm\,cm$, I hear "usually between 100 cm and 140 cm."

What's perhaps not intuitive is that you know more about the mean than you do about any particular measurement. The "standard error on the mean" goes like $\sigma/\sqrt N$, where $\sigma$ is the standard deviation of the distribution and $N$ is the number of samples that are included in the computation of the mean. For example, this dataset has $\sigma = 20\rm\,cm$ and $N=1000$, so the uncertainty on the mean is $\sigma/\sqrt N = 0.6\rm\,cm$. The actual mean I compute from these thousand data points is $(123.3 \pm 0.6)\rm\,cm$, which is totally consistent with the mean of 123.2 cm that I put in by hand.

To see a little more clearly the difference between the width of a distribution and the uncertainty on the mean, here are histograms of ten different sets of 1000 measurements each, generated the same way as the one above:

The mean of of each data set is represented with a fat blue dot. On the left, where you can see the entire distribution, you can just barely tell that not all of the means are the same. On the right, where only the means are shown, you can see that the uncertainty estimate $\sigma/\sqrt N = 0.6\rm\,cm$ looks like a good estimator of the uncertainty on the mean, since about two-thirds of the means are within one error bar of the correct value. This is like meta-statistics: doing statistics on the means and standard deviations of several data sets.

This is a general pattern with statistics: it makes more sense if you can actually play with some data where you already know some of the things you're interested in.