# What statistical test should I use?

I have two differential cross sections $d\sigma_{1,2}/dM$ ($M$ is some invariant mass) corresponding to different hypothesis. What I want is to calculate how well the appropriate experiment would do in distinguishing this two hypothesis. More precisely, I want to calculate the minimal luminosity the experiment should accumulate for being able to exclude the alternative hypothesis at specified confidence level.

The first thing that came to my mind was to use chi-squared test and calculate the expected value of test-statistic. But it has some constrains for the number of events in each bin since we have Poisson Distribution rather than Gaussian. So it's not very useful test for small data. I've also discovered that even if we have enough events in each bin the smaller number of bins gives better result. But it seems to me that we don't use all information when we have just two or three wide bins.

Next, I thought about using likelihood ratio test-statistic: $$X=\sum_{i=1}^N \left.\left(\frac{e^{-N^{alt}_i}(N^{alt}_i)^{d_i}}{d_i!}\right)\right/\left(\frac{e^{-N_i^{null}}{(N_i^{null}})^{d_i}}{d_i!}\right)$$ Where $N$ is number of bins, $N_i^{null}$ and $N_i^{alt}$ correspond to estimated number of events for null and alternative hypothesis respectively.

But there are two regions of $M$: in first region null hypothesis gives smaller number of events than alternative, in other region null hypothesis gives bigger number of events. It means that $X$ does not monotonically increase for more alternative-like data, so it's inconvenient. Of course I can cut off second region, but it's again a loss of information.

So is there optimal way to calculate such kind of things?

-
–  nibot Apr 4 '12 at 12:04
@nibot as I understand, KS-test works only for continuous distributions which is not the case. –  vitalik Apr 4 '12 at 13:13
–  dmckee Apr 4 '12 at 15:42
BTW--An alternate place for asking this kind of question is Cross Validated (the statistics Stack Exchange site). Upside of using stats is you're talking to statisticians. The downside is their language is perhaps a little different from that used by physicists. –  dmckee Apr 4 '12 at 15:45
@dmckee Thanks a lot! Heard about ROOT, but didn't really know about its capabilities. –  vitalik Apr 4 '12 at 16:45