What would you recognize as an absolutely fatal objection?
Let us brush aside the fact that it more or less is an assumption that experimental physics is doomed to fail to accurately reveal important features of the world and that science is doomed. These consequences make this question a better fit for Philosophy.SE than here; but never mind — let's tackle the philosophical question, and accept that it is entirely conceivable (if not very productive to put into practise) that the enterprise of modern physics is futile.
What we are basically discussing, here, is whether or not conscious human choices are constrained by local hidden variables in such a way, by a deterministic universe, that it tends to give rise to choices of measurements which reveal correlations which seem impossible for local hidden variables; or whether conscious human choices about what numbers we would choose to factor are constrained so that we only choose to factor the numbers which the hidden variables of the quantum system happen to be useful to factor. That is: that our brains — one of the most complicated uncontrolled systems that we know of — either has a vast supply of secret correlations with the world around us, or is so thoroughly prone to ingress of such secret information, that it serves to constantly thwart any attempt to discover the fact that the world around us is not random, nor involves any apparently non-local effects.
This is Cartesian paranoïa at its finest. In order for a deterministic physics to give rise to conscious agents such as ourselves, and at the same time spoof us with outcomes that seem routinely probabilistic, with reasonable convergence of frequencies of events to reliable averages, by directing us to think that we are choosing to make measurements for which the outcomes seem random but correlated in a way that defies expanation by local hidden variables — this seems to me to require not just a fine-tuned universe hospitable to life à la the anthropic principle, but a cosmological coincidence so enormous as to require an intelligent agency whose purpose is to deceive; to design ants just so that it could poke at anthills. While this position presents a solution to such classic theological conundrums as The Problem of Evil, does it really seem like a simpler hypothesis than just the idea that nature is stochastic, and a just little more complicated than we've ever had reason to believe before 1900 CE?
Consider this, as well: if we manage to build a large quantum computer, there is nothing to prevent us from setting up a computer so that it will multiply each pair of odd primes over 106 in some fixed order, and then re-factor each such product using Shor's algorithm. The only "choices" involved in the running of the computer, then, are the details of the design of the computer, the precise moment when we decide to turn it on, and the mathematical theory (running all the way down to number theory!) which underlies its operation. Failing mysterious failures of the computer, a cosmic conspiracy which tried to fool us into believing quantum mechanics would have to be responsible for the precise moment we turn the computer on, and the entire course of the development of mathematics and engineering leading to Shor's algorithm and the building of the quantum computer, to continue to fool us. Furthermore, these forces which determine our behaviour so precisely must somehow manage to avoid detection, either by making us constitutionally unable to realise that they're going on, or by giving rise to these developments in technology by a sudden cascade of events at the last second, even as our behaviour is being guided in other ways by the same hidden variables.
There is simply no reason, at present, to consider this a reasonable hypothesis. It is delicate to the extreme, and I cannot even begin to fathom how precisely the initial conditions would have to be tuned in order to make it true — especially in view of the fact that complex systems exhibit such instability that the "cosmic lie" would presumably be quickly revealed unless the initial conditions were set just so. Only someone who was absolutely committed to determinism and local realism, in preference to any sane and pragmatic approach to epistemology or ontology, would ever entertain it. But then, if the world is actually as they describe, we shouldn't blame them for it; after all, they have no more choice to believe it than I have to mention quite tangentially that 11×13 = 143.