Statistical significance in Z boson excess paper In a recent ATLAS paper, a excess of $3\sigma$ was reported from the Standard Model prediction in a search looking for a same-sign dilepton signal with an invariant mass around the Z peak. For brevity I discuss their results in the electron channel. 
Their results are shown in table 7 of the paper where they quoted an observed $16$ events with an expected background in $4.2 \pm 1.6$. The full table is shown below:

This result alone naively suggests a deviation from the SM of $7.4\sigma$!. However, instead of using this estimation ATLAS uses pseudo-experiments to test "how often such a deviation would occur from the SM". The results of this method are shown in table 11,

 They find a background of $8 ^{+4}_{-2}$ which is about $3\sigma $ away from their observation, $20$. 
I'd like to understand why this second step was needed? In essence they are disregarding their data-driven background estimation technique to estimate their discrepancy. However, the data-driven approach is the one that one would naively expect to be more precise.
Are they simply trying to be conservative or is there a good reason why the second approach is required?
 A: Let me begin with your naive estimation of the significance. Sixteen ($o=16$) events are observed over an expected background of $b=4.2\pm1.6$. I presume that you have calculated that
$$
\frac{o - b}{\delta b} \simeq 7.4,
$$
concluding that the excess has a significance of $7.4\sigma$. There are a two main mistakes in this interpretation of the data and calculation.
Firstly, in the background only hypothesis, we expect the number of observed events, $o$, to be Poisson distributed with expectation $b$. The error $\delta b=1.6$ is the uncertainty on the expectation of a Poisson distribution; it isn't the standard deviation of the number of observed events. Note, however, that because the background is estimated from a subsidiary counting experiment, $\delta b$ contains a systematic and statistical component.
Secondly, we don't know the distributions. Even if you did know that a result was $7.4$ standard deviations away from what we expected, you couldn't find out the significance of the excess. Standard deviations only specify a probability content if you specify a particular distribution. You don't know the distribution - and if you want to know the probability in the tail of a distribution, as you do for significances, you need to take care.
In summary, ATLAS don't appear to be doing things twice or altering their statistical procedure in light of big excess. They are calculating the significance once in the usual manner by Monte Carlo, taking into account full information about the uncertainties and their distributions, and the inherent Poissonian nature of their counting experiment.
This is a classic statistical problem that occurs in physics, sometimes referred to as the "on/off problem." If you point a detector at a source ("on") and measure a rate, then point your detector away from that source ("off"), how do you make a statistical inference about the effect of the source? See Cousins et al for a discussion.

A third point, $S^{95}$ indicate $95\%$ upper-limits on quantities, rather than quantities themselves. I wouldn't try to estimate the significance from these quantities (again you don't know the distributions etc), but it's not that surprising that it's closer to the result of the full calculation. The error in the expected $S^{95}$ is $95\%$ coverage in repeated pseudo-experiments - including the errors on the backgrounds and the Poissonian nature of the backgrounds. 
