# Jackknife estimation of variance very different from expected variance

I was rapidly introduced to the Jackknife procedure for data analysis and I stumbled upon a problem I'm not able to understand.

Let's consider a very simple case: the estimation of the average of a particular variable. When I compute the average:

$$\overline{x}=\frac{1}{N}\sum_{i=1}^{N}x_i \tag{1}$$ and the associated variance

$$\sigma^2=\frac{1}{N(N-1)}\sum_{i=1}^{N}(x_i-\overline{x})^2 \tag{2}$$

I get a certain value. When I run the jackknife algorithm, equations for clarity:

Set of data (uncorrelated): $x_{experimental\ data}=\{x_1,...,x_N\}$

n-th jackknife average from the subset without the n-th value:

$$\overline{x}_{n_{JK}}=\frac{1}{N-1}\sum_{i\ne n}x_i \tag{3}$$

Jackknife average:

$$\overline{x}_{JK}=\frac{1}{N}\sum_{i=1}^N\overline{x}_{i_{JK}} \tag{4}$$

Jackknife variance: $$\sigma^2_{JK}=\frac{N-1}{N}\sum_{i=1}^{N}(\overline{x}_{JK}-\overline{x}_{i_{JK}})^2 \tag{5}$$

I get from both (4) and (5) something identical or very similar to (1) and (2) when I have a small (<100000) number of samples, but increasing them, while the jackknife average (4) is still very similar to (1), now the jackknife variance (5) gets smaller and smaller compared to (2), soon becoming many orders of magnitude smaller. This happens also for more complicated functions, not just the plain average of something.

I don't know if I misanderstood the algorithm, if it's an error in the implementation or if this is exacly what I should expect from the jackknife variance.

In your case, $\bar x_{JK}$ and $\sigma_{JK}^2$ should be exactly equal to $\bar x$ and $\sigma$. You can show that by taking equations (4) and (5) and plugging in the Jackknife definitions (and the definition of $\bar x$) and doing some (perhaps ugly) algebra. Of course $\sigma_{JK}^2$ should become smaller with increasing $N$, but so should $\sigma^2$.

At first and second glance I couldn't spot any error in your equations, so maybe you're right and there's a bug in the implementation.

In general the Jackknife results (with or without bias correction) will be identical to the results of the usual formulas for mean and standard error as long as you're computing the plain average of some function $f$,

$$\bar f = \frac{1}{N} \sum_{i=1}^N f(x_i).$$

(Note that $x_i$ can be a vector or something else, too.)

In our collaboration we use to call these “primary observables”, but I don't know if that's standard terminology. So in these cases there's no point in going through the Jackknife procedure.

This holds for any function $f$, be it $x^5$, $\sin(x)$, $x/|x|$ or whatever. Also for linear combinations of primary observables the Jackknife results agree, since you can pull the linear combination into the average, e.g.

$$2 \bar f - 3 \bar g = \overline{2 f - 3 g}.$$

So in effect it's a primary observable itself.

Where Jackknife does make a difference is if you have a non-linear function of primary observables. (We call these “derived observables”.) An example is the Binder cumulant (see OP's comment), the ratio of the expectation value of $x^4$ and the square of the expectation value of $x^2$,

$$\frac{\langle x^4 \rangle}{\langle x^2 \rangle^2}.$$

(I'm using the angle brackets here to denote the actual mathematical expectation values in contrast to the average of a finite sample.)

A natural estimator would be

$$\hat\theta = \frac{\overline{x^4}}{\left( \overline{x^2} \right)^2} = \frac{\frac{1}{N} \sum_{i=1}^N x_i^4}{\left( \frac{1}{N} \sum_{i=1}^N x_i^2 \right)^2}.$$

$\overline{x^4}$ and $\overline{x^2}$ considered separately are primary observables, so you can estimate their standard errors in the usual way without Jackknife. To estimate the standard error of $\hat\theta$ you could then use simple error propagation,

$$\sigma_{\hat\theta}^2 = \left( \frac{\partial\hat\theta}{\partial\overline{x^4}} \right)^2 \cdot \sigma_{\overline{x^4}}^2 + \left( \frac{\partial\hat\theta}{\partial\overline{x^2}} \right)^2 \cdot \sigma_{\overline{x^2}}^2,$$

but this approach has several problems:

• The estimator $\hat\theta$ may be biased, i.e. for finite sample sizes $N$ it may yield results that are too large or too small on average. This is mostly overcome by the Jackknife method as it removes the part of the bias that goes with $1/N$. However, according to OP's comment and my own experimentation this doesn't seem to be a big issue here anyway and one can do without the bias correction.

• The error estimator $\sigma_{\hat\theta}$ effectively linearizes $\hat\theta$, because it uses only its first derivatives. It ignores contributions from higher-order derivatives. Sadly, Jackknife doesn't help here, either, but again the effect is small anyway.

• The most significant problem is that the error estimator ignores the correlation between $\overline{x^4}$ and $\overline{x^2}$. If you happen to have a sample where the average $x^2$ is larger than usual, the average of $x^4$ is probably larger, too. In the ratio, both deviations would cancel partially. So the actual error of $\hat\theta$ will be smaller than the result of simple error propagation. Jackknife automatically takes care of these correlations and should give a much more accurate estimate of the standard error.

Another potential problem is auto-correlation, i.e. correlation of the $x_i$ at different $i$, as it would arise in a typical Monte Carlo simulation based on Markov chains. This will make the error estimate smaller than the actual error. In contrast to the correlation of $\overline{x^4}$ and $\overline{x^2}$, Jackknife doesn't take auto-correlation into account at all. Instead one often uses binning to deal with auto-correlation, i.e. one divides the $x_i$ into consecutive groups of equal size, takes the average of each group and analyzes the averages instead of the original $x_i$. Alternatively one can use a variant of Jackknife where consecutive groups of $x_i$ are removed from the sample instead of a single element at a time, but I couldn't find a good reference for that.

• found the bug. Anyway I didn't even know that the Jackknife is intended to be used for linear functions. Apart from my trivial example I was requested to use it for example to calculate a Binder coefficient, which is absolutely not linear. In that case it's normal to get a very different result between JKvariance and usual variance? – Mattz Jul 20 '18 at 0:31
• @Mattz I don't know how to calculate a Binder coefficient (do you have a link for that?), but I guess in general all bets are off and the results can be arbitrarily different. You can still compare it to results from other resampling methods like bootstrapping. By the way, does the Binder coefficient as you compute it depend only on the sample average, like f(x̅), or is it a function of all individual data points, f(x_1, x_2,..., x_N)? – Sunfoil Jul 22 '18 at 15:34
• It goes under the name of Binder cumulant, it's something proportional to $\frac{\overline{x^4}}{ (\overline{x^2})^2}$ . x_i's are just measurements of the magnetization of a spin system at different montecarlo times by the way. In that case I've read that a computational algorithm for the error is the way to go because of the non-linearity. By the way, just for the record, I've found a quite good book explaining this stuff: "everything you wanted to know abut data analysis and fitting but were afraid to ask"-P.Young. The bias related to JK is said to be negligible most of the times – Mattz Jul 22 '18 at 15:54
• @Mattz I have extended my answer quite a bit. Probably you don't need that any more, but maybe someone else finds it useful. I've experimented a little, and I think you're right. The bias is negligible in that case. The relevant point seems to be to take into account the correlation between $\overline{x^2}$ and $\overline{x^4}$. Also, thanks for the book tip! Is it the same as this Arxiv article by Young? – Sunfoil Aug 12 '18 at 20:23
• And I think my statement that the Jackknife results would be different for non-linear functions was misleading. What I meant was non-linear functions of averages like $(\bar x)^4$. If the non-linear function is inside the average, e.g. $\overline{x^4}$, Jackknife doesn't make a difference. – Sunfoil Aug 12 '18 at 20:24