My motivations for asking this question are philosophical but I think this is a question that is best answered by the physics community.

There is a problem in the philosophy of science called the 'underdetermination' problem. There are two versions.

  1. One is the idea that for any set of data, theory choice is underdetermined. There could be multiple theories which explain the data in the same way and we don't yet have the ability or the data required to choose between them on completely empirical grounds.

  2. This is the idea that two theories could predict exactly the same thing in all situations but use different unobservable entities to explain it.

As far as I know (early on in the process at least) Lorentzian interpretation of SR vs Einstein's interpretation of SR could be compared in this way. Both made similar or the same predictions but many favoured SR in terms of its simplicity, not having to posit the existence of an unobservable ether. (Obviously later the empirical differences were evaluated and the ether theory had to be modified to account for its failed predictions and slowly lost favour but these empirical considerations aren't really the point of this post).

There are three main questions from this:

  1. What are the main criteria that physicists approach when deciding between two theories that make the same predictions or which have yet to be decided up on experimentally?

  2. Is it even possible to construct empirically identical theories which posit different unobservable entities to explain phenomena? Is there any reason to believe that this is impossible?

  3. In practice, does this ever actually happen?

P.s. It seems as though the conflict between the Lorentzian and Einsteinian interpretations (at least early on) was an example of this but I may have some details about the history of these ideas wrong. As far as I know, the Lorentzian interpretation was able to be modified to fit with experimental failures but was later rejected on the grounds that it was much more complex (I assume this means it required many additional assumptions?) and therefore was rejected by the community at large.

It does seem however that both interpretations posit some unobservable thing. In the Lorentzian case, it seems as though he outright assumes the existence of an ether. In the Einsteinian case, there is some unobservable warping of space time. I say this warping is unobservable because you can only observe the effects of it, though this may also be wrong. So if anyone wants to clear up any misunderstanding I have about the important differences between the ether and Einstein's idea of spacetime warping, please do. Similarly, if I'm wrong to say that only the effects of spacetime warping can be observed, please correct me. Although, at this moment in time, I don't know what it would mean to directly observe the warping of spacetime since spacetime is the canvas on which we make observations??

The tag 'popular-science' is here because I'm not studied in theoretical physics very much. I have just completed A-level physics (somewhere in between high school and university by US standards) and wish to study theoretical physics at university which is why I find the problem so interesting and I used the Einstein vs Lorentzian argument as the example for this question because during my study I did a report about this argument and about simple derivations of Lorentz transformations etc... so it's an argument that I'm more familiar with than others.

I suppose a similar debate would have been during he original development of kinetic theory as kinetic theorists posited the existence of atoms which were unobservable at the time in order to explain certain phenomena which could have been explained with some other kind of unobservable (maybe, I don't know)?

  • $\begingroup$ What is a "warping of spacetime"? $\endgroup$ – WillO Jun 15 '18 at 18:52
  • 1
    $\begingroup$ At first, the anther was a medium, in which electromagnetic waves oscillated. This was not supported by the observation until Lorentz used his equations. Then Einstein noticed that with the Lorentz equations the existence of the anther became moot and made no difference, so it was simpler to just drop it. And then in general relativity spacetime was curved. Well, philosophically (what many mathematicians pretending to be physicists don't get), an empty "place" cannot be "curved". The curvature is a property that fills this "place". And so the anther is back in a form of the metric tensor. $\endgroup$ – safesphere Jun 15 '18 at 19:47
  • 2
    $\begingroup$ I think your Einstein vs Lorentz is a good example. For the little it's worth, I've always thought physicists rated theories by (1) fewness of ad hoc hypotheses, (2) ability to make surprising predictions (3) beauty, usually mathematical. Ernest Rutherford (perhaps not a profound philosopher of science) said (or so it said on the flyleaf of my ancient A-Level electricity textbook) "These fundamental things have got to be simple." $\endgroup$ – Philip Wood Jun 15 '18 at 19:47
  • 1
    $\begingroup$ Physics is a science of observable quantities. If there is no observable, then there can be no measurement. $\endgroup$ – Peter Diehr Jun 15 '18 at 20:23
  • 1
    $\begingroup$ @PeterDiehr Are you saying there is no dark matter or dark energy? Neither has been observed any more than the aether. Both are just "rescue" concepts invented to provide a life support for the theories that fail to explain the actual observations. However, both are endorsed and funded by the establishment. $\endgroup$ – safesphere Jun 16 '18 at 6:41

To address,

What are the main criteria that physicists approach when deciding between two theories that make the same predictions or which have yet to be decided up on experimentally?

first and foremost it should be noted, without experimental evidence favouring one, the community will likely continue developments to some extent on all viable competing theories, naturally.

If there is no experimental support available, two significant factors are relied upon by theorists:

  1. Aesthetic appeal: Does the theory require constraints to be imposed by hand or do they arise naturally? Does the theory possess notable symmetries or mathematical properties?
  2. Theoretical consistency: Is the theory in agreement with other established theories, either exactly or in some limit? Is the theory self-consistent?

The latter is easier to demonstrate quantitatively, while the former relies to some extent on a subjective view of mathematical beauty or appeal, though constraints put in ad hoc by hand for example would be a potential red flag.

An example of a 'beautiful theory' is certainly general relativity, in the sense provided by Lovelock's theorem: for $d=4$, up to second derivatives in $g_{\mu\nu}$, the choice $\mathcal L = \sqrt{|g|}\mathcal R$ is the only option.

As for consistency, take your favourite tree level scattering amplitude from quantum electrodynamics, and in the classical limit, it should correspond to the cross section for the process evaluated using non-relativistic quantum mechanics.

| cite | improve this answer | |
  • $\begingroup$ That makes plenty of sense. I think it's possible that 1. could be carried out by assessing the 'number' of assumptions in theories. It seems to be the case that a theory packed with ad hoc hypotheses would have many foundational assumptions. This is probably the case because if some (bad) theoretical physicist has devised a theory and desperately wants to save it from experimental disconfirmation, he would have to add assumptions to create new predictions and maintain others so that the theory didn't completely change? Would you agree with this? $\endgroup$ – Joe Lee-Doktor Jun 15 '18 at 22:09
  • $\begingroup$ I'm obviously not experienced in theoretical physics research (at all) but from what I understand the basis of 'theory building' is in making certain physical assumptions, understanding their consequences and then assessing their predictions experimentally. $\endgroup$ – Joe Lee-Doktor Jun 15 '18 at 22:10

The answers above are fairly comprehensive, so I will just add a point by way of analogy which I think gives an intuition for why physicists (and scientists in general) prefer simpler theories and tend to be more inclined to believe they are somehow fundamentally correct if they give the right predictions.

It is common in science to have to perform "curve fitting", where one has many sets of measurements (to give a concrete example, let's say we have 50 measurements of the force between two newly discovered particles and the distance between them), generally with some noise and measurement error, and must determine how well they agree with some model. In our example, we might have a model of force where $F(r) = \frac{a}{r^2}$. Here we have a single parameter, $a$, so we have to answer two questions - which value of $a$ makes the model best fit our measured data, and once we've found it, how well does it fit? Usually it won't fit perfectly even if the model was correct because the measurements are never perfect, but generally the better the fit the more confident we can be in the model. We may compare two models by asking which can be made to better fit data.

However we must be careful with this line of reasoning, because of a phenomena called "overfitting". Let's say someone else comes along and claims that the force is described by a 100th degree polynomial, i.e. $F(r) = a_{100}r^{100} + a_{99}r^{99} + a_{98}r^{98} + ... + a_2r^2 + a_1r^1 + a_0r^0$. As it turns out, given any set of $n$ pairs of measurements, we can perfectly fit that data with an $n$th degree polynomial, so that means our 100th degree polynomial can be perfectly fitted to our 50 measurements.

But obviously this doesn't mean the model is correct, in the case of our two new particles, because we could have always fit those data-points regardless of the underlying physics. This is known as "overfitting". This is also why most judgements of how well a model agrees with data don't just account for how closely it fits, but how many degrees of freedom the model had to fiddle with.

In general when discussing physical theories we may not have lots of easy to count numerical parameters. Whether one theory or the other has greater or fewer metaphorical parameters is non-obvious, and two theories where one seems more complicated may turn out to be mathematically, or even conceptually identical. Thus physicists are forced to rely on an intuitive sense of "elegance", a gut feeling that says a theory is "simple", "inevitable", that it is either right or wrong and can't be easily modified to fit observations that disagree slightly. I believe Feynman once summed up the feeling when he said something like "You can't put imperfections on a perfect thing, you have to come up with a new perfect thing instead". Perhaps one day we will figure out how to quantify this idea, but for now we're stuck with our guts.

| cite | improve this answer | |

To further answer

Is it even possible to construct empirically identical theories which posit different unobservable entities to explain phenomena?

Another example of the underdetermination problem is in the interpretation of quantum mechanics. Here you have several competing theories of how nature behaves on small scales, which are all purportedly consistent with the predictions of QM (at least so far...). These include the many-worlds, pilot wave, transactional, and Copenhagen interpretations. These theories offer dramatically different visions of the universe, but by construction they have the same experimental predictions.

So how does one choose an interpretation? As mentioned previously, aesthetic appeal, simplicity, consistency, and utility for predictions are often important. But since QM requires such a large departure from our everyday intuition, I think that someone will also choose an interpretation which offends their sensibilities the least. For example, if you really want determinism but don't mind non-locality (i.e. spooky action at a distance), choose Bohmian pilot-waves. But if you'd rather save locality at the expense of mild retrocausality, go with the transactional interpretation. If you don't like to think about it and would prefer to select a theory based on familiarity and ubiquity, choose Copenhagen.

Personally, I like how the transactional interpretation gives you the Born probability rule "for free", in the sense that it is not needed as a postulate (as it is for every other interpretation); rather, it is "justified". This enhances the transactional interpretation's simplicity and, in my mind, aesthetic appeal.

| cite | improve this answer | |
  • $\begingroup$ In that case I think it's certainly true that simplicity should reign supreme. And, like you said, perhaps philosophical investigation of the assumptions made by each interpretation can uncover internal inconsistencies with some. Or at least we can try to discover which interpretations make the most 'plausible' assumptions. And I assume you're taking simplicity = aesthetic appeal. Which most people seem to. Otherwise, I don't think aesthetic appeal is a rational criteria with which to judge a theory/interpretation. I think people should be more careful with using the terminology of aesthetics $\endgroup$ – Joe Lee-Doktor Jun 16 '18 at 8:08

If you want a 'cop out' answer to your number 2, there's always 'chance', that could cause your empirical observations to 'look similar' to one another. Of course, repeated measurements would make such less likely by each measurement. But it does mean it's possible! (meaning there's no reason to believe it's impossible!).

Secondly, to take a bit of a philosophical / tongue-in-cheek approach to this. What you're asking is in sort to ask for a limit to our 'imagination', since wouldn't you agree that this is what's used when coming up with 'the possible reasons for the unobservable'.

I also don't find it too hard to believe in there being 2 equally flawed interpretations that both coincide with one another (in their observational effects).

lo and behold, some bad gravity 'interpretations' examples!

  1. gravity is a small particle that when it collides with objects, it doesn't slow down (it passes through the object), and deposits 'negative kinetic energy' (for some reason?, further theorycrafting required).

  2. gravity is a fully connected graph of invisible & untouchable & unbreakable rubber bands, with its 'relaxed' state at the 'origin point of the universe' (where the big bang happened :p ), everything is connected! (even photons, since we can see gravity bending light!). Also... the bands get weaker the more they are stretched... even to the point of pushing away after a certain length?

^ those are pretty bad theories, but... you can imagine that they would produce 'similar observable outcomes' (at least... to a certain point, and the cracks start to show... I guess... or maybe the patched rubber bands theory is the way it works :p).

In the end, I guess we can only really apply the occam's razor principle as a loose measure of believability for a theroy / idea.

| cite | improve this answer | |
  • $\begingroup$ Yeah, I think you're right that any number of theories can be proposed to explain/describe some phenomena. Something I wonder, though, is whether or not you can really have two sets of identical empirical predictions (logical consequences of some assumptions) from different sets of assumptions. Whether or not any 'set' of consequences has some minimum number of assumptions that led to it. It seems at least intuitive to think that, for example, if you need some set of 3 assumptions which entail some set of consequences, there is no other set of 3 that couldn't entail those same consequences? $\endgroup$ – Joe Lee-Doktor Jun 15 '18 at 22:29
  • $\begingroup$ @JoeLee-Doktor One of the best examples of this situation is actually the foundation of general relativity. The assumption "I'm in a gravitational field" and the assumption "I'm in an accelerating reference frame" make identical predictions. $\endgroup$ – probably_someone Jun 16 '18 at 0:16
  • 1
    $\begingroup$ @probably_someone It seems like those assumptions are trivially different though. Like, they make the same predictions but isn't that because the 'gravitational field' is just the label for the accelerated reference frame we're referring to. Or, better yet, 'gravitational fields' are a subset of accelerating reference frames. So the assumption "I'm in a gravitational field" entails the fact that "I'm in an accelerating reference frame"? $\endgroup$ – Joe Lee-Doktor Jun 16 '18 at 8:03
  • $\begingroup$ @probably_someone Also the assumption that space is bent or curved $\endgroup$ – Bill Alsept Jun 16 '18 at 16:14
  • 1
    $\begingroup$ @GoodQuestions You seem to be confused on what a physical theory is. It is not a word salad as in your examples, but a mathematical predictive model that matches observational measurements very well. For example, QED matches experimental data to better than one part in a trillion. To compare two theories, you need to have two different mathematical models that predict the same experimental measurements. One example is the Newtonian gravity and general relativity that predict very close results for weak gravitational fields. $\endgroup$ – safesphere Jun 16 '18 at 16:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.