What Data Set(s) Would Be Best For Head-To-Head Unified Field Theory Model Selection? I'm seriously considering establishing a prize for unified field theories in-line with my history of prize awards, such as the fusion legislation I drafted with a cofoudner of the AEC's program, the Bowery Award for Amateur Rocketry (that led the Ansari X-Prize) and The Hutter Prize for Lossless Compression of Human Knowledge.
The rules would be identical to The Hutter Prize rules with one exception:
Replace the enwik9 Wikipedia corpus with physics data.
The value of this prize would be the rigorous and objective model selection criteria:
Solomonoff Induction
Hutter wrote a paper titled "A Complete Theory of Everything (will be subjective)" in which he explains the applicability of Solomonoff Induction to physics, including what it takes to select among any set of UFT's, including those with large numbers of free parameters, such as string "theory".  I scare-quote "theory" in the case of "string theory" (and would for any purported UFT with free parameters) because the, somewhat sloppy, vernacular phrase "physical theory" has been taken to mean what the more accurate phrase "physical model" means.  In other words, until a physical interpretation is coupled to a theory it remains in the domain of metaphysics if not pure mathematics.   Any theory with free parameters remains indeterminate, hence "not even wrong" as there is no interpretation.
A reductio ad absurdum of the view that model selection can take place in the presence of free parameters:
Any "parameter" of unspecified precision, can, via arithmetic coding, provide an unspecified amount of information.  A theory with such a free parameter can populate it with an infinite precision floating point number and then, via arithmetic coding, expand it to cover any observations whatsoever.
Expanding on how this would work:
Each school of theoretic thought would create an executable archive of the physics corpus.  That executable archive program would have to run within the resource limitations appropriate to the UFT competition.
For instance, purpose served by The Hutter Prize is about a week of 1 CPU's time using less than 10GB of RAM is permitted.  But other language modeling "competitions" invest millions of dollars of compute resources.  Whatever the compute resource choice, the resulting file must match, bit for bit, the original physics corpus.
The shortest program at any given time would be awarded money from the prize purse according to how much it reduced the size from the prior winner's benchmark.
The physics corpus should be adequate to the prize's purpose.  For example, The Hutter Prize's corpus, enwik9 (1 billion byte snapshot of Wikipedia) is adequate to its purpose of modeling natural language knowledge.
A critical but inadequate corpus would be The NIST's Fundamental Physical Constants "Value" column encoded as variable precision floating point.  An obvious addition to the fundamental constants would be the Ionization Energy column from the NIST Atomic Spectra Database Ionization Energies Data.
I looked around and failed to find any pre-existing such corpus, or list of corpora for testing theories against each other in a rigorously fair manner.
What set of physical data would be adequate as a model selection tool?
 A: Comparing theories in this way is not likely to be relevant.
Consider the actual candidates for theory of everything. We have string theory, which is uniquely compelling as a step beyond field theory, but whose empirical application requires searching an ill-understood googol-sized landscape of possibilities for worlds that appear to resemble our own, and then calculating the detailed properties of physics in each such world. Machine learning is being used to conduct the search for qualitatively promising worlds, but the second step, of detailed calculation, requires much more progress in fields like algebraic geometry.
Then we have all the theories that aren't string theory: like, loop quantum gravity coupled to a grand unified field theory, or celebrity theories of everything like those due to Lisi or Weinstein, or dozens of similar but unheralded proposals that have been posted to arxiv. None of these theories can calculate everything either - indeed, most of them can't calculate much at all. They're all "works in progress".
On the other hand, you have extensions of the standard model that include new fields and make slightly different predictions. Like the standard model, these are field theories with numerous free parameters. Conceivably you could rank such theories by the number of bits required to specify them, and the number of bits of the standard model parameters that they would explain; but such theories generally make entirely new predictions too (e.g. proton decay), and how they fare on that front is more important for their ultimate credibility.
A: I think there are some features of the problem of predicting various measured quantities from principles, from a computer science perspective, that make it unlike predicting text from a natural language model. Here are a few (there is some overlap between them):

*

*We never know the ground truth in physics. For natural language processing, there is a clear goal: the text of wikipedia is a snapshot of how people write English today, and can serve as a ground truth for a NLP model. In physics, we don't know many crucial experimental facts that would be needed to distinguish between candidate models at the Planck scale. Is there supersymmetry broken at some scale? If so, which scale? What is the scattering cross section of dark matter with neutrinos? What is the fourth digit of the quartic self-coupling constant of the Higgs Boson? We don't know these quantities, but surely they will be needed to distinguish candidate models. Perhaps there are errors in the current knowledge of fundamental constants, so that a theory that matches the list too well will actually turn out not to be a good one in the end.


*Computational complexity is not a proxy for truth. While we want the simplest model that will fit the data, there is no way to put an absolute bound on the computational resources needed for an accurate model. A Newtonian gravity simulation would take far fewer resources than a full GR simulation, but to accurately model gravitational waveforms we surely need the latter. Such a simulation pushes the boundaries of what is capable with supercomputers today, but this should not be taken as a point against GR, compared to a phenomenological fit that could reproduce observed gravitational-wave data. In fact, simulating a quantum mechanical system with $N$ qubits takes a very large number (exponential in $N$) of classical resources.


*Not all insights are data driven. MOND does an excellent job of fitting galaxy rotation curves. However, by itself, we know it is not a complete theory because it is not relativistic. There are proposals to make MOND relativistic, and I don't want to get into an argument about whether MOND is correct or not, but the point is that there are criteria beyond comparing to data that guide us to a correct description. I suspect a potpourri of different phenomenological models that fit different data in the corpus could do a very good job fitting a set of physics measurements, but would certainly not be acceptable as a fundamental theory of physics.


*The value of physics is to be predictive outside of the scope of the original data. Data-driven approaches like machine learning work well when the training set covers a representative range of cases that cover likely data that will be seen in the future. Said differently, statistical models are good at interpolation but poor at extrapolation. On the other hand, the value of fundamental physics is precisely to make predictions about situations that have never been seen before. Nothing in our everyday experience could make us think it would be reasonable to predict the state of the Universe 3 minutes after the Big Bang, for example. But taking our theories of GR and nuclear physics and extrapolating them in a conservative way, we are able to model this process and make predictions that are later confirmed by experiment. Testing theories by how they extrapolate into new areas is a crucial part of the development of physics, but is something that would not be possible in this kind of approach.
