15
$\begingroup$

What is the difference between the Kolmogorov-Smirnov test and the Chi-squared test?

When should we use one instead of the other?

I was reading this article, and I got confused a lot.

It is hard to get a physics-related answer in the Cross Validated SE section, so this is why I am asking here.

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ I think this should probably really be on cross-validated, so just as a comment: I'm not really sure how to give a physics-centered answer, but I'll try. You use a chi-square test to test the goodness of a fit, for example of a curve to data points, and when your goal is to minimize statistical errors. You use a KS test when you have a hypothesis (probability density) and some data, and want to see if it is plausible that the data was sampled randomly from the PDF (=is consistent) or not. Looking for something in that direction? $\endgroup$
    – jdm
    Commented Apr 10, 2014 at 8:29
  • $\begingroup$ Thanks. Yes, I think we are on the good way. So, what is the difference between a hypothesis and a model? $\endgroup$
    – Py-ser
    Commented Apr 10, 2014 at 8:36
  • $\begingroup$ That you should really take to cross-validated if you want an accurate definition :-). We often blur the distinction between both terms in everyday work. What we often mean is we have some theory which gives us a formula, or a Monte Carlo program, etc., and that spits out an expected, "ideal" result in form of a probability density.... To your question, a hypothesis is an assumption (either true or false). A model is a theory where you have several "knobs" (parameters) to turn. A hypothesis could be "model A is true with x=0.5", and then you could test if that is consistent with data. $\endgroup$
    – jdm
    Commented Apr 10, 2014 at 8:55
  • $\begingroup$ Despite what OP thinks, would Cross Validated be a better home for this question? $\endgroup$
    – Qmechanic ♦
    Commented Apr 10, 2014 at 12:41
  • $\begingroup$ Every time I try to get a clear answer in Cross Validated is really really hard hard work... $\endgroup$
    – Py-ser
    Commented Apr 11, 2014 at 1:49

1 Answer 1

Reset to default
17
$\begingroup$

A chi-squared test is used to compare binned data (e.g. a histogram) with another set of binned data or the predictions of a model binned in the same way.

A K-S test is applied to unbinned data to compare the cumulative frequency of two distributions or compare a cumulative frequency against a model prediction of a cumulative frequency.

Both chi-squared and K-S will give a probability of rejecting the null hypothesis. Artificially binning data loses information so should be avoided if possible. On the other hand the chi-squared statistic does give useful shortcuts if you are trying to model the parameters that describe a set of data and the uncertainties on those parameters. The K-S test should not really be used if there are adjustable parameters which are being optimised to fit the data.

Specific trivial example. I measure the height of 1000 people. Let's say they're all between 1.5m and 2m feet. I have a model I wish to test that says the distribution is Gaussian with a mean of 1.76m and a dispersion (sigma) of 0.1m.

So, how do I test whether this model well represents the data? One approach is to construct the cumulative distribution of heights and then compare it against the cumulative normal distribution described using a KS test. However, an alternative would be to put the data into say 5cm bins and then find the chi-squared statistic compared with the model. Both of these would give you a probability of rejecting the null hypothesis. For such a purpose though I would favour the K-S test, because binning the data takes away some information.

On the other hand maybe your hypothesis is that the distribution is normal and you want to find what the mean and dispersion are. In which case you can't use the K-S test, that's not what it is for. However you can minimise chi-squared to find the best fitting parameters using the binned data. A caveat here would be that when dealing with frequencies, chi-squared should not be used when you have small numbers per bin (say less than 9), because Poisson statistics become important. In these instances there are alternatives like the "Cash statistic".

I suppose at some level data are always binned. But when doing the K-S test there is usually only one object in each bin!

NB: The K-S test is not uniformly sensitive to differences in distributions at all values. It is most sensitive to differences in the median and very insensitive to differences in the tails of the distributions. A better test all round is the Anderson-Darling test is generally a better way of comparing two sample distributions.

Improve this answer
$\endgroup$
12
  • $\begingroup$ Thanks. Anyway, I am probably missing the point about binned data. We always have binned data, because of the instrument resolution, isn't it? $\endgroup$
    – Py-ser
    Commented Apr 11, 2014 at 2:02
  • $\begingroup$ More added above in an edit. $\endgroup$
    – ProfRob
    Commented Apr 11, 2014 at 10:00
  • $\begingroup$ Could you please explain on chi-squared statistic does give useful shortcuts if you are trying to model the parameters that describe a set of data and the uncertainties on those parameters. Did you mean we can use the parameter deduced from the dataset in Chi-squared test? $\endgroup$
    – Akira
    Commented Mar 15, 2020 at 15:57
  • 1
    $\begingroup$ @Crush_on_You These lectures on $\chi^2$ fitting seem ok to me. astronomy.swin.edu.au/~cblake/StatsLecture3.pdf $\endgroup$
    – ProfRob
    Commented Mar 15, 2020 at 18:21
  • 1
    $\begingroup$ @Crush_on_You The minimum chi-squared can be used as a means of rejecting the hypothesis that the optimised model is a good fit to the data. $\endgroup$
    – ProfRob
    Commented Mar 18, 2020 at 7:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.