# Correction factor for standard deviation

I have conducted an experiment, from which I obtained a finite amount of values for the gravitational acceleration. The amount of values ranges from 6 up to ~150 values.

I have the unbiased sample variance and the (approximated) standard deviation defined as follows:

\begin{align*} s^2 &= \frac{1}{n-1} \sum\limits_{i=1}^n \left(x_i - \bar{x}\right)^2\\ s &= \sqrt{s^2} \end{align*} ($n$ is the total number of values, $x_i$ is the i-th value, $\bar{x}$ is the arithmetic mean,
note that I forgot the factor 1/(n-1) in a previous revision of this post)

I have been given the following formula for the mean error for few measured values: $$\Delta x = \frac{t}{\sqrt{n}}s$$ Where $t$ is to be looked up in the next table:

+-------------------------+-------------------+
| Number of single values | correction factor |
+-------------------------+-------------------+
| 5                       | 1.15              |
| 6                       | 1.11              |
| 8                       | 1.08              |
| 10                      | 1.06              |
| ... (cut)               | ... (cut)         |
| more                    | 1.00              |
+-------------------------+-------------------+


Question: Which correction factor should I take for 7 single values?
Shall I round up to the next available correction factor?

My teacher is currently unavailable, I therefore hope this question is objectively answerable with the information given.

• Would Cross Validated be a better home for this question? – Qmechanic Nov 2 '14 at 13:44
• @Qmechanic Possibly, although the questions seems to be of a much higher level there. Nonetheless, I'd be happy if you could move the Q :) – visitor Nov 2 '14 at 13:58

You are trying to estimate the mean of population that your data come from, and you are assuming a normal distribution for the data.

However, you do not know the true variance of the population, and you need to use the data to estimate the variance as well as the mean. Therefore, you don't get quite as much "information" out of your experiment as you would have hoped: you have to use up some of the "discovery power" to estimate another parameter of the distribution (i.e. variance/standard deviation) before you can begin to do the "real work" of estimating the mean. Thus, the estimated means come from a distribution that is a little bit broader than the normal - the Fisher t-Distribution of $n-1$ degrees of freedom. As your sample size $n\to\infty$, the difference between the Fisher and Normal distributions gets smaller and smaller.

You could, therefore, look up your value from tables of the Fisher t distribution. However, with your given values, you could do a linear interpolation and get a pretty good result. That is, $7$ is half way between $6$ and $8$, so the mean of $1.11$ and $1.08$ i.e. $1.095$ would be a good value to use. Notice that the decrease between $8$ and $10$ is less than decrease betweem $6$ and $8$; this means your curve is "concave up" (its rate of decrease decreasing), so that $1.095$ will be a slight overestimate of the true value. This is good. It means you will not be assuming any more accuracy than your experiment can give you.

• Thanks! The linear interpolation sounds easier (and more likely to be what my teacher wants, I think). Did you cut off your last sentence by any chance? – visitor Nov 2 '14 at 13:08
• @visitor No, it was a dangling piece of text scribbled down as I wrote the answer. Fixed now. – WetSavannaAnimal Nov 2 '14 at 13:34

Just one more look at this... given the numbers, you could fit a curve through the data. If you decided to include one additional point on either side of the missing value (4 points total) you could fit an exact cubic, and the interpolated value would be 1.100; if instead you did an (inexact) quadratic fit through the four points you would get 1.093. A linear interpolation through the two adjacent points gives 1.095 (it would not be a good idea to fit a straight line through the four points as there is obvious curvature). Finally, you could do the interpolation by computing the curvature from the four points, then applying that curvature as a correction to the linear interpolation between the two middle points.

Here is what these "fits" to the data would actually look like: where I plot $n$ along the horizontal, and the quadratic/cubic fit to the four correction values given along the vertical.

Of course since the correction values were only given to 2 sf it is hard to justify getting too wrapped around the axle about the difference - the cubic fit is clearly a poor representation of the underlying data (see how the slope changes sign...).

With that said, I think that WetSavannaAnimal's approach (linear interpolation, accepting that the value is probably a little bit conservative) is the right one.

Or you can go back to theory - but it's tricky and while it may be "statistically correct" to fret about these differences, it is practically almost never worth your while (except to get a good grade which gets you into the right university which gets you a good job which ...)

• Thanks for your take on this! Is there any justification why a cubic or quadrativ curve is correct? If one wanted to be pedantic, one could argue that the value for 7 can be anything (e.g. 1000). I suppose a proof would require the underlying formulae of the values, which I admittedly don't understand. – visitor Nov 2 '14 at 16:00
• A functional approximation is given here: en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation but the numbers it gives do not match those in your table. – Rob Jeffries Nov 2 '14 at 16:51
• "Justification" is a big word - but continuous functions can be represented by a Taylor series and for sufficiently small deviations from a known value that means either a linear, quadratic, cubic or higher order approximation is appropriate. In this case inspection of the four points says linear is not good enough - so you end up trying higher order. Can't go above 3rd (only four points) and cubic seems to over fit (see how slope changes direction). That makes parabolic interpolation "pretty good". – Floris Nov 3 '14 at 3:17

I agree with @WetSavannaAnimal as far as interpolating to get the correction factor goes.

Alternatively you could go and find out what the correction factor for 7 values actually is.

However, the formulae you originally quoted for the unbiased sample variance and the sample standard deviation were incorrect.

The formula for unbiased variance is $$s^2 = \frac{1}{n-1} \sum_{i}^{n} (x_i - \bar{x})^2$$

When you take the square root of this you get a biased estimator of $s$, the standard deviation. This all discussed here, where a formula is given for a correction factor to $s$ that is appropriate if you know that your data are normally distributed, which usually, you don't.

I am puzzled by your correction factors. Were you told that the data followed something other than a normal distribution? There are approximations that can be used to calculate the corrections for a normal distribution. In the form that you define it, the value for $n=7$ is 1.0423 and the correction factor for other $n$ values do not match the ones in your table.

?

• You're right, I totally forgot the $\frac{1}{n-1}$ factor when I wrote the question. In fact, I have only been given that table and the formulae. How does one know which distribution one's data follow? – visitor Nov 2 '14 at 15:52
• @visitor Well in practice you often don't. A normal distribution is the standard assumption unless you know otherwise. So I can't immediately see where your table numbers come from, or if I have made a mistake in my interpretation of the correction factors. – Rob Jeffries Nov 2 '14 at 16:38
• The document which contains my table numbers does not mention the source itself. It only mentions the statistical certainty which is 68,3 %. May I ask where your correction factor of 1.0423 (for n=7) comes from? – visitor Nov 2 '14 at 17:24
• It comes from the second link I put in my post. – Rob Jeffries Nov 2 '14 at 17:52
• Read carefully. Your $t$ is $c(4)^{-1}$ I believe. – Rob Jeffries Nov 2 '14 at 18:02