Model Selection in Physics My understanding of the scientific method is that it can be summarized in the following steps that don't need to be executed in any particular order:


*

*Make conjectures & hypotheses (i.e. develop models and theories)


*Make predictions from them


*Carry out experiments and/or collect data


*Test and possibly embrace the new theories / models IF:

*

*the data fit model predictions more accurately than alternative theories

*the new theory is not more complex than other plausible (fitting) alternatives




In  statistics and computational learning we often encounter a similar tension between goodness of fit and model complexity when comparing models that aim to explain the data. To do this, we rely on formal model selection methods such as the Bayes factor and its approximations (e.g. AIC, BIC, deviance information criterion, etc.), and often use measures of model complexity and validation to decide what particular model to embrace.
My question:
Are there any examples of these or similar frameworks in Physics that are used to compare theories?  In other words, are there any information theoretic frameworks researched / used in Physics that study this particular tradeoff between model accuracy and complexity to specifically inform theory selection?
 A: I will comment and reorder your list, as a retired experimental particle physicist.

   Make conjectures & hypotheses (theory)


Preexisting successful theories  with their postulates and strict mathematical models .  After all physics started before Newton.

   Make predictions from this theory


Use the theory to predict behaviors in experiments carried out currently, to confirm/validate preexisting theories.

   Carry out experiments and observations


Surprise , surprise experiment does not fit preexisting theory. Head scratching of experimentalists, fever by theoreticians.
Example: radioactivity needed special relativity and quantum mechanics to be modeled theoretically, and the observations existed long before the theories,  experimental data forced the need for new theories.
New theories appear:

   Test and embrace the new theory if
        the data fit the predictions more accurately than alternative theories
        the new theory is not more complex than other plausible alternatives


No. Test and embrace the new theories for the new region of validity and make sure that the old theories can mathematically be shown to emerge from the new. For example: statistical mechanics the new theory, was shown to have as emergent theory Thermodynamics, an elegant mathematical model working well in its region of validity long before statistical mechanics was formulated.
Then design experiments that may show diversions from the current model leading to a deeper theoretical understanding, as now with the LHC the standard model is being tested/validated and everybody is holding their breath that a discrepancy will be found leading to the need of hypothesized higher theories.
Physics does not progress by having a theorist propose a brand new  model to be checked. This has lead to a lot of crackpot proposals, with people not understanding why they are not treated as the new Einstein.
Einstein built upon the previous theories, certainly thinking outside the box, but his theories were based on the previous ones extending them for new regions of validity, and the join between the older theories and General relativity and special relativity is smooth and computable.  No  general  "out with the old in with the new" works for the scientific method in physics.
And to answer about statistical methods, all statistical methods are used in evaluating the goodness of fit of data to theories. Example the recent Higgs search and discovery at the LHC. 
Also to keep in mind that in particle physics now, the specialty of theorist and experimentalist is necessary as the amount of knowledge and expertise needed in each branch is enormous. One symptom is the 3000 physicists signing the experimental papers of the LHC.
