Jackknife estimation of variance very different from expected variance I was rapidly introduced to the Jackknife procedure for data analysis and I stumbled upon a problem I'm not able to understand.
Let's consider a very simple case: the estimation of the average of a particular variable. When I compute the average: 
$$\overline{x}=\frac{1}{N}\sum_{i=1}^{N}x_i  \tag{1}$$
and the associated variance
$$\sigma^2=\frac{1}{N(N-1)}\sum_{i=1}^{N}(x_i-\overline{x})^2 \tag{2}$$
I get a certain value. When I run the jackknife algorithm, equations for clarity:
Set of data (uncorrelated):
$x_{experimental\ data}=\{x_1,...,x_N\}$
n-th jackknife average from the subset without the n-th value:
$$\overline{x}_{n_{JK}}=\frac{1}{N-1}\sum_{i\ne n}x_i  \tag{3}$$
Jackknife average:
$$\overline{x}_{JK}=\frac{1}{N}\sum_{i=1}^N\overline{x}_{i_{JK}}  \tag{4}$$
Jackknife variance: $$\sigma^2_{JK}=\frac{N-1}{N}\sum_{i=1}^{N}(\overline{x}_{JK}-\overline{x}_{i_{JK}})^2 \tag{5}$$
I get from both (4) and (5) something identical or very similar to (1) and (2)
when I have a small (<100000) number of samples, but increasing them, while the jackknife average (4) is still very similar to (1), now the jackknife variance (5) gets smaller and smaller compared to (2), soon becoming many orders of magnitude smaller. This happens also for more complicated functions, not just the plain average of something.
I don't know if I misanderstood the algorithm, if it's an error in the implementation or if this is exacly what I should expect from the jackknife variance.
 A: In your case, $\bar x_{JK}$ and $\sigma_{JK}^2$ should be exactly
equal to $\bar x$ and $\sigma$.  You can show that by taking equations
(4) and (5) and plugging in the Jackknife definitions (and the
definition of $\bar x$) and doing some (perhaps ugly) algebra.  Of
course $\sigma_{JK}^2$ should become smaller with increasing $N$, but
so should $\sigma^2$.
At first and second glance I couldn't spot any error in your
equations, so maybe you're right and there's a bug in the
implementation.

In general the Jackknife results (with or without bias correction)
will be identical to the results of the usual formulas for mean and
standard error as long as you're computing the plain average of some
function $f$,
$$ \bar f = \frac{1}{N} \sum_{i=1}^N f(x_i). $$
(Note that $x_i$ can be a vector or something else, too.)
In our collaboration we use to call these “primary observables”, but I
don't know if that's standard terminology.  So in these cases there's
no point in going through the Jackknife procedure.
This holds for any function $f$, be it $x^5$, $\sin(x)$, $x/|x|$ or
whatever.  Also for linear combinations of primary observables the
Jackknife results agree, since you can pull the linear combination
into the average, e.g.
$$ 2 \bar f - 3 \bar g = \overline{2 f - 3 g}. $$
So in effect it's a primary observable itself.

Where Jackknife does make a difference is if you have a non-linear
function of primary observables.  (We call these “derived
observables”.)  An example is the Binder cumulant (see OP's comment),
the ratio of the expectation value of $x^4$ and the square of the
expectation value of $x^2$,
$$ \frac{\langle x^4 \rangle}{\langle x^2 \rangle^2}. $$
(I'm using the angle brackets here to denote the actual mathematical
expectation values in contrast to the average of a finite sample.)
A natural estimator would be
$$ \hat\theta = \frac{\overline{x^4}}{\left( \overline{x^2} \right)^2} = \frac{\frac{1}{N} \sum_{i=1}^N x_i^4}{\left( \frac{1}{N} \sum_{i=1}^N x_i^2 \right)^2}. $$
$\overline{x^4}$ and $\overline{x^2}$ considered separately are
primary observables, so you can estimate their standard errors in the
usual way without Jackknife.  To estimate the standard error of
$\hat\theta$ you could then use simple error propagation,
$$ \sigma_{\hat\theta}^2 = \left( \frac{\partial\hat\theta}{\partial\overline{x^4}} \right)^2 \cdot \sigma_{\overline{x^4}}^2 + \left( \frac{\partial\hat\theta}{\partial\overline{x^2}} \right)^2 \cdot \sigma_{\overline{x^2}}^2, $$
but this approach has several problems:


*

*The estimator $\hat\theta$ may be biased, i.e. for finite sample
sizes $N$ it may yield results that are too large or too small on
average.  This is mostly overcome by the Jackknife method as it
removes the part of the bias that goes with $1/N$.  However,
according to OP's comment and my own experimentation this doesn't
seem to be a big issue here anyway and one can do without the bias
correction.

*The error estimator $\sigma_{\hat\theta}$ effectively linearizes
$\hat\theta$, because it uses only its first derivatives.  It
ignores contributions from higher-order derivatives.  Sadly,
Jackknife doesn't help here, either, but again the effect is small
anyway.

*The most significant problem is that the error estimator ignores the
correlation between $\overline{x^4}$ and $\overline{x^2}$.  If you
happen to have a sample where the average $x^2$ is larger than
usual, the average of $x^4$ is probably larger, too.  In the ratio,
both deviations would cancel partially.  So the actual error of
$\hat\theta$ will be smaller than the result of simple error
propagation.  Jackknife automatically takes care of these
correlations and should give a much more accurate estimate of the
standard error.
Another potential problem is auto-correlation, i.e. correlation of
the $x_i$ at different $i$, as it would arise in a typical Monte Carlo
simulation based on Markov chains.  This will make the error estimate
smaller than the actual error.  In contrast to the correlation of
$\overline{x^4}$ and $\overline{x^2}$, Jackknife doesn't take
auto-correlation into account at all.  Instead one often uses
binning to deal with auto-correlation, i.e. one divides the $x_i$
into consecutive groups of equal size, takes the average of each group
and analyzes the averages instead of the original $x_i$.
Alternatively one can use a variant of Jackknife where consecutive
groups of $x_i$ are removed from the sample instead of a single
element at a time, but I couldn't find a good reference for that.
