# Does frequentism require exponentially many trials in some cases?

Frequentism is the philosophy that probabilities are statistical in the sense that they give the limiting frequency ratios of outcomes as the number of trials is large enough. For tiny probabilities like exponentially small probabilities, would this require exponentially many trials?

I know, I know, you are thinking if the probability of an outcome is exponentially small, then we won't expect it to happen in any trial if the number of trials is some reasonable practical number. This overlooks the situation where say the possible outcomes are some long strings of characters taken from some alphabet in the computer science sense, and the probability distribution is such that the probability for any particular string is exponentially small. In such a case, if someone were to specify some specific string in advance, and the number of trials is some realistic number, no one would expect to find that string in any trial outcome. On the other hand, if you were to look at the trial outcomes, they would all correspond to strings which if chosen a priori, would be considered practically improbable. In practice, what experimenters do in such a situation is conduct statistical randomness tests upon the strings obtained, but does this take us out of the realm of frequentism?

Can frequentism be saved in such a case?

• I think a Bayesian vs. Frequentism debate fits better to math.stackexchange. The number of trials for a given probability can be calculated, but is this really the point here. Given a model and some data there are different ways to analyze the two together, both ways are valid but use a different interpretation of the outcome. Commented Nov 18, 2011 at 17:33
• "exponentially". I do not think that word means what you think it means. When you say "exponentially small," do you just mean "very small?" Commented Nov 18, 2011 at 17:44
• @ColinK - in the example given, the probability of finding a specific string of length N decreases exponentially with N. So it seems reasonable to refer to this as an exponentially small probability.
– user2963
Commented Nov 18, 2011 at 17:57
• If a given string looks unlikely, it may just mean that we haven't understood the process that created it. The play "Hamlet" wasn't unlikely for whoever wrote it. A typical failing of statistical analysis is to assume statistical independence where there are in fact high-order correlations. The probability of writing "Hamlet" near the word "Claudius" is greater in Hamlet than they are in random text, but capturing all such structure in terms of a low-level concept like correlations is tricky. Commented Nov 18, 2011 at 18:54

One way to address this Question is to respond that frequentism is not the standard interpretation of probability in Physics, so it doesn't have to be saved. See Section 3.3 of this Stanford Encyclopedia of Philosophy page, for example, for an account of the woes of frequentism.

Instead, some form of Statistical hypothesis testing is the de facto standard. That's why one hears that an experiment is consistent or not consistent with a given (probabilistic) Physical theory at "the 3 sigma level", or "at 5 sigmas", etc., which is a concise way of reporting levels of statistical significance. If experimental results fall outside 5 sigmas, that is very strong but not absolutely compelling motivation to introduce a different probabilistic model that makes predictions that are more consistent with the statistics of experimental results.

Bayesian analysis fits into this understanding as a formulaic way to modify a given statistical hypothesis on the basis of experimental data, on the assumption that the statistical model is broadly correct, it's just the parameters that have to be determined. If the structure of a probabilistic model is not in sympathy with the data, however, Bayesian rules will not suggest, for example, a different space-time structure that will work better.