The Law of Large Numbers does not say as much as you seem to think it says. There is a great deal of misunderstanding about this. Suppose that the probability of a coin's toss being heads is one half. The Law of Large Numbers says nothing at all about what the asymptotic results will be. It only says what the probability is that the asymptotic results will be $X$. That the asymptotic result of the ratio of heads will be unity has a probability of zero, for example. But probability zero does not mean physically impossible, and hence the Law does not say that the asymptotic result will not be unity. The great English analyst Littlewood explained this clearly in a famous maths club talk, later published in his collection of essays, A Mathematician's Miscellany, entitled "The Dilemma of Probability". Kolmogorov himself said essentially the same thing, in print. For more references and a discussion (oh, and other people have also referred to the play Rosencrantz and Guildenstern Are Dead, by Tom Stoppard), see my "Logic of Physical Probability Assertions", http://arxiv.org/abs/quant-ph/0508059 , and prof. Jan von Plato's important article cited there.
The excerpt from Littlewood is as follows:
Mathematics \dots has no grip on the real world;
if probability is to deal with the real world it must contain elements
outside mathematics, the \it meaning\rm\ of «probability» must relate
to the real world; and there must be one or more «primitive» propositions
about the real world, from which we can then proceed deductively (i.e., mathematically).
We will suppose
(as we may by lumping several primitive propositions together)
that there is just one primitive proposition,
the «probability axiom», and
we will call it «$A$» for short.
\dots the «real» probability problem;
what are the axiom $A$ and the meaning of «probability» to
be, and how can we justify $A$?
It will be instructive to consider the attempt called the «frequency theory». It is
natural to believe that if (with the natural reservations) an act like throwing
a die is repeated $n$ times the proportion of 6's will, with certainty,
tend to a limit, $p$ say, as $n \rightarrow \infty$. (Attempts are made to
sublimate the limit into some Pickwickian sense---«limit» in inverted commas.
But either you mean the ordinary limit, or else you have the problem
of explaing how «limit» behaves, and you are no further. You do not make an
illegitimate conception legitimate by putting it into inverted commas.) If we
take this proposition as $A$ we can at least settle off-hand the other
problem, of the meaning of probability, we can define its measure for
the event in question to be the number $p$. But for the rest this $A$
takes us nowhere. Suppose we throw 1000 times and wish to know what to expect.
Is 1000 large enough for the convergence to have got under way, and how far?
$A$ does not say. We have, then, to add to it something about the rate
of convergence.
Now an $A$ cannot assert a certainty about a
particular number $n$ of throws, such as «the proportion of 6's will
certainly be within $p\pm\epsilon$ for large enough $n$ (the largeness
depending on $\epsilon$)».
It can only say « the proportion will lie between $p\pm\epsilon$ with at
least such and such probability (depending on $\epsilon$ and $n_o$) whenever
$n>n_o$.» The vicious circle is apparent. We have not merely failed to justify a workable
$A$; we have failed even to state one which would work if its
truth were granted.
http://www.library.uu.nl/digiarchief/dip/diss/1957294/c4.pdf
and
http://philsci-archive.pitt.edu/archive/00000367/00/ergodic.ps
are two reviews of prof. von Plato's ergodic theory of probability, which itself is not on-line.
