# Statistical significance in Z boson excess paper

In a recent ATLAS paper, a excess of $3\sigma$ was reported from the Standard Model prediction in a search looking for a same-sign dilepton signal with an invariant mass around the Z peak. For brevity I discuss their results in the electron channel.

Their results are shown in table 7 of the paper where they quoted an observed $16$ events with an expected background in $4.2 \pm 1.6$. The full table is shown below:

This result alone naively suggests a deviation from the SM of $7.4\sigma$!. However, instead of using this estimation ATLAS uses pseudo-experiments to test "how often such a deviation would occur from the SM". The results of this method are shown in table 11, They find a background of $8 ^{+4}_{-2}$ which is about $3\sigma$ away from their observation, $20$.

I'd like to understand why this second step was needed? In essence they are disregarding their data-driven background estimation technique to estimate their discrepancy. However, the data-driven approach is the one that one would naively expect to be more precise!

My feeling is that they are worried they were scared of a $7\sigma$ deviation and thought they might have underestimated their systematics so to be on the safe side they used MC to estimate their significance. Are they simply trying to be very conservative or is there a good reason why the second approach is required?

Let me begin with your naive estimation of the significance. Sixteen ($o=16$) events are observed over an expected background of $b=4.2\pm1.6$. I presume that you have calculated that $$\frac{o - b}{\delta b} \simeq 7.4,$$ concluding that the excess has a significance of $7.4\sigma$. There are a two main mistakes in this interpretation of the data and calculation.

Firstly, in the background only hypothesis, we expect the number of observed events, $o$, to be Poisson distributed with expectation $b$. The error $\delta b=1.6$ is the uncertainty on the expectation of a Poisson distribution; it isn't the standard deviation of the number of observed events. Note, however, that because the background is estimated from a subsidiary counting experiment, $\delta b$ contains a systematic and statistical component.

Secondly, we don't know the distributions. Even if you did know that a result was $7.4$ standard deviations away from what we expected, you couldn't find out the significance of the excess. Standard deviations only specify a probability content if you specify a particular distribution. You don't know the distribution - and if you want to know the probability in the tail of a distribution, as you do for significances, you need to take care.

In summary, ATLAS don't appear to be doing things twice or altering their statistical procedure in light of big excess. They are calculating the significance once in the usual manner by Monte Carlo, taking into account full information about the uncertainties and their distributions, and the inherent Poissonian nature of their counting experiment.

This is a classic statistical problem that occurs in physics, sometimes referred to as the "on/off problem." If you point a detector at a source ("on") and measure a rate, then point your detector away from that source ("off"), how do you make a statistical inference about the effect of the source? See Cousins et al for a discussion.

A third point, $S^{95}$ indicate $95\%$ upper-limits on quantities, rather than quantities themselves. I wouldn't try to estimate the significance from these quantities (again you don't know the distributions etc), but it's not that surprising that it's closer to the result of the full calculation. The error in the expected $S^{95}$ is $95\%$ coverage in repeated pseudo-experiments - including the errors on the backgrounds and the Poissonian nature of the backgrounds.

• I'm pretty unsure about your statement on the Poisson uncertainty. In the paper, the caption mentions that the uncertainties displayed in OP's table contains both statistical and systematic uncertainties. So I'd expected the uncertainties quoted in the table to be the RMS of the full PDF (a product of Poisson and as many Gaussians as necessary for the many nuisance parameters). – Michaël Ughetto Mar 25 '15 at 17:07
• @MichaëlUghetto the background estimates are data-driven. they contain a systematic and statistical uncertainty. – innisfree Mar 26 '15 at 7:48
• Thanks for your response. I'm still trying to understand this. What I find confusing is that based on what you are saying, it seems that the data-driven background estimate is effectively useless since its not used to estimate the discrepancies? – JeffDror Mar 27 '15 at 4:11