5
$\begingroup$

What is the difference between the Kolmogorov-Smirnov test and the Chi-squared test?

When should we use one instead of the other?

I was reading this article, and I got confused a lot.

It is hard to get a physics-related answer in the Cross Validated SE section, so this is why I am asking here.

$\endgroup$
  • 1
    $\begingroup$ I think this should probably really be on cross-validated, so just as a comment: I'm not really sure how to give a physics-centered answer, but I'll try. You use a chi-square test to test the goodness of a fit, for example of a curve to data points, and when your goal is to minimize statistical errors. You use a KS test when you have a hypothesis (probability density) and some data, and want to see if it is plausible that the data was sampled randomly from the PDF (=is consistent) or not. Looking for something in that direction? $\endgroup$ – jdm Apr 10 '14 at 8:29
  • $\begingroup$ Thanks. Yes, I think we are on the good way. So, what is the difference between a hypothesis and a model? $\endgroup$ – Py-ser Apr 10 '14 at 8:36
  • $\begingroup$ That you should really take to cross-validated if you want an accurate definition :-). We often blur the distinction between both terms in everyday work. What we often mean is we have some theory which gives us a formula, or a Monte Carlo program, etc., and that spits out an expected, "ideal" result in form of a probability density.... To your question, a hypothesis is an assumption (either true or false). A model is a theory where you have several "knobs" (parameters) to turn. A hypothesis could be "model A is true with x=0.5", and then you could test if that is consistent with data. $\endgroup$ – jdm Apr 10 '14 at 8:55
  • $\begingroup$ Despite what OP thinks, would Cross Validated be a better home for this question? $\endgroup$ – Qmechanic Apr 10 '14 at 12:41
  • $\begingroup$ Every time I try to get a clear answer in Cross Validated is really really hard hard work... $\endgroup$ – Py-ser Apr 11 '14 at 1:49
6
$\begingroup$

A chi-squared test is used to compare binned data (e.g. a histogram) with another set of binned data or the predictions of a model binned in the same way.

A K-S test is applied to unbinned data to compare the cumulative frequency of two distributions or compare a cumulative frequency against a model prediction of a cumulative frequency.

Both chi-squared and K-S will give a probability of rejecting the null hypothesis. Artificially binning data loses information so should be avoided if possible. On the other hand the chi-squared statistic does give useful shortcuts if you are trying to model the parameters that describe a set of data and the uncertainties on those parameters. The K-S test should not really be used if there are adjustable parameters which are being optimised to fit the data.

Specific trivial example. I measure the height of 1000 people. Let's say they're all between 1.5m and 2m feet. I have a model I wish to test that says the distribution is Gaussian with a mean of 1.76m and a dispersion (sigma) of 0.1m.

So, how do I test whether this model well represents the data? One approach is to construct the cumulative distribution of heights and then compare it against the cumulative normal distribution described using a KS test. However, an alternative would be to put the data into say 5cm bins and then find the chi-squared statistic compared with the model. Both of these would give you a probability of rejecting the null hypothesis. For such a purpose though I would favour the K-S test, because binning the data takes away some information.

On the other hand maybe your hypothesis is that the distribution is normal and you want to find what the mean and dispersion are. In which case you can't use the K-S test, that's not what it is for. However you can minimise chi-squared to find the best fitting parameters using the binned data. A caveat here would be that when dealing with frequencies, chi-squared should not be used when you have small numbers per bin (say less than 9), because Poisson statistics become important. In these instances there are alternatives like the "Cash statistic".

I suppose at some level data are always binned. But when doing the K-S test there is usually only one object in each bin!

$\endgroup$
  • $\begingroup$ Thanks. Anyway, I am probably missing the point about binned data. We always have binned data, because of the instrument resolution, isn't it? $\endgroup$ – Py-ser Apr 11 '14 at 2:02
  • $\begingroup$ More added above in an edit. $\endgroup$ – Rob Jeffries Apr 11 '14 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.