I have already posted this problem here, closed for asking too many questions. I will reproduce the introduction and contextualization here, and try to ask a more specific one that I hope will be acceptable.

I have performed an analysis of the exclusion(discovery) perspectives for a BSM particle at the LHC.

The considered signal are 2 new diagrams for the process $pp \to X$ (obeying some here irrelevant kinematic cuts and ignoring jets). I implemented the new physics model and generated pseudo-experiments for hundreds (let's say N) of points in a bidimensional parameter space -- let's say it is generated by $(M,\theta)$. I also generated the same number of events of (irreducible) background, for the null hypothesis which can also produce the exact same process. Chosen as discriminant observable was the invariant mass $m_{||}$ of certain $n$-body product of the diagrams. I generated histograms of the number of events with relation to $m_{||}$ for each of the N points within the parameter space, and for the background.

To reject the signal containing hypothesis at each point, "I have chosen a 95% confidence level" (the quotes are because I don't know how correctly am I using these concepts, but I will not keep writing them). The confidence level in each point was motivated by the following starting expression

$$ \tag{1} \text{Confidence level} =1-\prod_{k}\frac{P(n<n_{0k} \;|\; \mu_{sk}+\mu_{bk})}{P(n<n_{0k}\; |\;\mu_{bk})}, $$
where $P(n \;|\; \lambda)$ is a Poisson distribution for variable $n$ and rate $\lambda$, which, in my case and for specified luminosity, is simply the expected number of events of the respective kind. $k$ labels the bins of $m_{||k}$, and $s_k$($b_k$) is the number of signal(background) events in the respective bin. $\mu_{sk}+\mu_{bk}$ is the expected number of events in the alternate hypothesis (signal + background) and $\mu_{bk}$ the number expected in the null (only background) hypothesis. $n$ is the number of events and $n_0$ defines an "upper limit of the confidence interval" to be chosen.

The final expression is obtained by "estimating" the signal(background) rate by the number of events predicted by the respective pseudo-experiment; and by fixing the "upper limit of the confidence interval" by the expected number of background events. That is

$$ \mu_{sk} \to s_k, \qquad \mu_{bk} \to b_k, \qquad n_{0k} \to b_k. $$

The result plugging this into Eq. (1) is

$$ \tag{2} \text{Confidence level}=1-\prod_{k}\left[ \frac{e^{-s_k}\sum\limits_{n=0}^{b_k} \frac{(s_k+b_k)^n}{n!}}{\sum\limits_{n=0}^{b_k} \frac{b_k^n}{n!}}\right], $$

Having obtained a Confidence level for each point in the space of parameters, I extrapolated the discrete result to the continuous plane and plotted contours of fixed $\text{CL}=95\%$.

What I did was validated by people with expertise, which makes me have a reasonable belief that (despite being obviously not sophisticated at all) it is, in principle, acceptable. I, however, do not truly understand it.

I am trying to learn this and acquire a formal enough understanding in order to perform more complex and professional analysis through these notes, and have its definitions and notions in mind.

The question is: is this a CL, $\text{CL}_S$ method or none of them? A descriptive and detailed answer would be very much appreciated since, since I have little, at most, understanding of what is happening here.

  • 1
    $\begingroup$ I encourage you to make your question more concise: e.g., your history, this question's history, and the stuff about $pp\to X$ are probably irrelevant $\endgroup$
    – innisfree
    May 22, 2020 at 4:52
  • 2
    $\begingroup$ Does this answer your question? Data analysis on particle physics: what am I formally doing in this specific case? $\endgroup$
    – innisfree
    May 22, 2020 at 5:05
  • $\begingroup$ Would Cross Validated be a better home for this question? $\endgroup$
    – Qmechanic
    May 22, 2020 at 16:56
  • $\begingroup$ @Qmechanic I have thought about that, but I think that, in general, the answer I'm looking for can be best provided by someone with a specific physical experience with these statistical concepts and I'm not familiar with how likely it is to be found there. It would not bother me if it was transferred there, though. $\endgroup$
    – GaloisFan
    May 22, 2020 at 19:19

1 Answer 1


What you are using is the Helene formula. (Also known as "the old PDG method"). It is the same as CLs when you are dealing with numbers of events (as you are here) rather than a likelihood analysis which also includes shape information.

For the frequentist CL you use your Equation 1 without the denominator. The. modification is an attempt to include the knowledge that if an observed number is small, then the background must have fluctuated downwards and you feel the limit set on the signal should not be so strict.

It does not have the correct frequentist properties (it over covers) but it can be deduced from a Bayesian approach with a uniform prior.

Use of CLs, including this version, is statistically hard to defend but nevertheless widespread.

  • $\begingroup$ Thank you very much for your answer! I have one request and a follow up question, if it would be convenient for you to answer. The question is: what would be a more defensible statistical approach to limit testing? And the request: can you indicate a reference in which I could understand the facts you stated about my method? $\endgroup$
    – GaloisFan
    May 22, 2020 at 15:30
  • $\begingroup$ You can legitimately use either CL or CLs provided you say what you are doing. CL will be more acceptable to statisticians and CLS to particle physicists. For details see for example Luca Lista's talk indico.in2p3.fr/event/6315/contributions/37805/attachments/… around slide 25 $\endgroup$ May 23, 2020 at 7:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.