What is a probabilistic physical theory? What do we mean when we say that we have a probabilistic theory of some phenomenon?
Of course, we know from experience that probabilistic theories "work", in the sense that they can (somehow) be used to make predictions about the world, they can be considered to be refuted under appropriate circumstances and they generally appear to be subject to the same kinds of principles that govern other kinds of explanations of the world. The Ising model predicts the ferromagnetic phase transition, scattering amplitude computations of quantum field theories predict the rates of transition between different quantum states, and I can make impressively sharp predictions of the ensemble properties of a long sequence of coin tosses by using results such as the central limit theorem. Regardless, there seem to be a foundational problem at the center of the whole enterprise of probabilistic theorizing - the construction of what is sometimes called "an interpretation of the probability calculus" in the philosophical literature, which to me seems to be an insurmountable problem.
A probabilistic theory comes equipped with an event space and a probability measure attached to it, both of which are fixed by the theory in some manner. However, the probability measure occupies a strictly epiphenomenal position relative to what actually happens. Deterministic theories have the feature that they forbid some class of events from happening - for instance, the second law of thermodynamics forbids the flow of heat from a cold object to a hot object in an isolated system. The probabilistic component in a theory has no such character, even in principle. Even if we observed an event of zero probability, formally this would not be enough to reject the theory; since a set of zero probability measure need not be empty. (This raises the question of, for instance, whether a pure quantum state in some energy eigenstate could ever be measured to be outside of that eigenstate - is this merely an event of probability $ 0 $, or is it in fact forbidden?)
The legitimacy of using probabilistic theories then rests on the implicit assumption that events of zero (or sufficiently small) probability are in some sense negligible. However, it's not clear why we should believe this as a prior axiom. There are certainly other types of sets we might consider to be "negligible" - for instance, if we are doing probability theory on a Polish space, the collection of meager sets and the collection of null measure sets are both in some sense "negligible", but these notions are in fact perpendicular to each other: $ [0, 1] $ can be written as the union of a meager set and a set of null measure. This result forces us to make a choice as to which class of sets we will neglect, or otherwise we will end up neglecting the whole space $ [0, 1] $!
Moreover, ergodic theorems (such as the law of large numbers) which link spatial averages to temporal averages don't help us here, even if we use versions of them with explicit estimates of errors (like the central limit theorem), because these estimates only hold with a probability $ 1 - \varepsilon $ for some small $ \varepsilon > 0 $, and even in the infinite limit they hold with probability $ 1 $, and we're back to the problems I discussed above. So while these theorems can allow one to use some hypothesis test to reject the theory as per the frequentist approach, for the theory to have any predictive power at all this hypothesis test has to be put inside the theory.
The alternative is to adopt a Bayesian approach, in which case the function of a probabilistic theory becomes purely normative - it informs us about how some agent with a given expected utility should act. I certainly don't conceive of the theory of quantum mechanics as fundamentally being a prescription for how humans should act, so this approach seems to simply define the problem out of existence and is wholly unsatisfying. Why should we even accept this view of decision theory when we have given no fundamental justification for the use of probabilities to start with?
I apologize for the length of the post, but I would like to hear what other people have to say about this problem.
 A: I don't think that probabilistic physical theories are so different from the other ones from the point of view of experimental falsifiability.
Indeed, even if your theory is deterministic, measurements are subject to uncertainties: there may be an error in the setup, human error, some external noise, etc. If your non-probabilistic theory states that something should not occur (say, neutrinos traveling faster than light) and you do observe such an event, then the first thing you do is look for such errors. Indeed, if the theory is well established, then this is by far the most likely explanation.
Now, if your probabilistic theory predicts that some event has probability very close to $1$ and you do not observe it, then you are certainly at least as justified to consider rejecting the theory as in the deterministic case above.
The central point is that the violation of the theory must be observable in some setting reproducibly. If the anomaly occurs just once, then you cannot conclude anything, because you'll not be able to exclude possible errors. That the theory is deterministic or probabilistic does not really matter.
Finally, concerning events with $0$ (or tiny) probability: such events occur constantly. If you throw a fair coin a million times, the probability of observing any particular sequence is $2^{-1000000}$, that is, zero for all practical purposes. But, of course, if you do the experiment then one such sequence will occur. The point is not that they do not occur, it is that you cannot predict their occurrence. In this example, there is a $\sim 0$ probability that if you choose a sequence before the experiment, the precise sequence you chose occurs. And if you managed to do that, then I would consider that something fishy is going on: the coin is not fair, or you are throwing it in a very special way, etc.
A: I am not sure that the conundrum in the question (and clarified in the comments) has really to do with probabilistic theory. In my view the culprit is actually the fact that we assign physical quantities to be described by values in the mathematical entity '$\mathbb{R}$'. This gives rise to all sorts of paradoxes. I'll try to explain with an example.
If we drop an (uncooked) spaghetti on the floor, we will likely use the very useful assumption that the angle that the spaghetti makes with the North Pole can be described by  $\theta\in [0, 2\pi)\subset \mathbb{R}$. Now according to the mathematical rules of probability and the concept of continuum, the probability of any given angle is zero. Yet every time we throw a spaghetti on the floor an angle is chosen (they all happen). What is the reason for this phenomenon?
If we think about it long enough the problem is that we are implicitly assuming that the nature of physical space can be described by the mathematical quantity $\mathbb{R}$. This is actually an heresy if you think about it. The mathematical object $\mathbb{R}$ has nothing to do with the physical world. It's a purely mathematical concept, that can give rise to crazy paradoxes like Banach-Tarski paradox (you can cut a sphere into a finite number of pieces and glue them back together in order to form two spheres).
If you really think about measuring the 'spaghetti angle' with a physical device, you will encounter problems when the required  precision reaches the level of atoms, as then the point where the spaghetti ends become fuzzy. But the problem as you see is about the nature of space itself. This problem has to do with renormalization of quantum field theories, string theories and all that stuff but this is another story.
A: There are two types of probabilistic physical theory (it is not particularly important whether one takes a Bayesian approach. A Bayesian understanding of probability is helpful, because it is clearer and more accurate, but it does not much change the essentials of the mathematics).
First, in classical probability theory, outcomes are determined by unknown variables. In principle, if we have a probability destribution for the unknowns, this will give us the probability distribution for the outcomes.
Second, in quantum theory, outcomes are not deterministic. There are no unknown variables generating quantum probabilities. It should be immediately obvious that this will not give us the same mathematical probability theory as a deterministic theory. It is rather less obvious that it will give us exactly the probability theory described by quantum mechanics. Nonetheless it does. I have explained the maths in The Hilbert space of conditional clauses.
