Your question is expressing a muddle about the measurement problem in quantum mechanics. You ask "is there any fundamental demonstration that the state of the system cannot have been determined before the measurement, yet according to the probabilistic rules of quantum mechanics". This is a bit unclearly asked, I think, but I take it you want to know if quantum behaviour is like classical probability, where one gets probablistic outcomes too. The answer is "yes and no". The quantum predictions for the outcomes of a given complete scenario, including whatever measurement apparatus is involved, give perfect ordinary probabilities which add to $1$ and behave in all respects in the way we expect for probability. On the other hand, these predictions are not statements about a quantum system alone; they are statements about the outcome of an interaction between the quantum system under discussion and whatever larger apparatus is involved---the apparatus commonly called "measuring apparatus".
Let's unpack this a little.
The standard way to formulate quantum mechanics asserts that any given system evolves according the Schrodinger's equation. This means that if some system is prepared and then evolves, then the state at some later time is fully determined by the theory, before any measurements are carried out, but this does not resolve all the issues to do with measurement.
By solving Schrodinger's equation one finds that before a measurement the state of the system is
$$
|{\psi(t)}\rangle = {\cal T} \exp\left( -i \int_0^t \hat{H} {\rm d}t/\hbar \right) |\psi(0)\rangle
$$
where $|\psi(0)\rangle$ is the state prepared at some initial time and
$\cal T$ is an operator called time-ordering operator, or Dyson time-ordering operator, which is required when the Hamiltonian is time-dependent (to be precise, when the Hamiltonian at one time does not commute with the Hamiltonian at another).
I have written this admittedly abstract answer in order to make it clear that the state of the system is completely determined before any measurement is made.
However, the interaction commonly called 'measurement' typically involves an interaction of the system with a complex apparatus, typically involving some sort of permanent change such as the avalanche in a photodetector or something like that. Tracking the details of such an avalanche leads one into the metaphysical puzzles surrounding measurement and interpretation of quantum theory. But for most results of interest we can avoid those metaphysical puzzles by asserting that the outcome of such an interaction is captured by a collapse postulate. So we note which are the states $| M_n \rangle$ which form the basis which a given type of measurement distinguishes in a stable way, and we just assert that the system state changes during the measurement so as to go to one of those states, with a probability given by
$$
\mbox{Prob[obtain $M_n$, given state $\psi(t)$]} = \left| \langle M_n | \psi(t) \rangle \right|^2
$$
This brings out the issues I mentioned in my opening paragraph. For, the state $|\psi(t)\rangle$ is determined, before the measurement, but this does not suffice to determine the result of each run of such an experiment, except in that it furnishes the above probabilities.
The essence of the issue here can be expressed as saying that, given a fixed type of measuring apparatus---the apparatus that is physically present in any given scenario---the quantum state $|\psi(t)\rangle$ is not so much the "state of the system" as a convenient way to furnish the probabilities. It is those probabilities which are the actual prediction of the theory, and the basis states accompanying them.
The projection $| \psi(t) \rangle \rightarrow | M_n \rangle$ is often presented in discussions as if it is a physical process (called 'collapse') undergone by the system, but one does not absolutely have to see it that way. Rather one can say that the physical world evolves between states that are the stable outcomes of irreversible processes, and the mathematical apparatus of quantum theory yields the probabilities for moving between these states. (This type of interpretation is close to the one called Copenhagen Interpretation, but it tries to be specific about what constitutes measurement by making an appeal to irreversibility).
If one sees $| \psi(t) \rangle \rightarrow | M_n \rangle$ as a physical process then one can assert that it takes place at any time between preparation and measurement---that is, at any moment during the time interval when ordinarily we say the evolution is unitary. To be precise, one would assert
$$
|{\psi(t_c)}\rangle \rightarrow {\cal T} \exp\left( -i \int_{t_m}^{t_c} \hat{H} {\rm d}t/\hbar \right) |M_n\rangle
$$
where $t_c$ is the moment assigned (rather arbitrarily) to this collapse, and $t_m$ is the time later on when one deems that the whole process is over. Thus in this way of looking at it, the system is deemed to collapse to whatever state is the one that evolves into the one observed later on as the outcome of the measurement. Quantum mechanics is not normally presented this way, but since it has the same outcomes and same probabilities it is equivalent to the more common interpretation where we suppose the collapse happens at the last minute. (But again, one need not regard the collapse as a physical process at all; see above).
