Dirac once said that he was mainly guided by mathematical beauty more than anything else in his discovery of the famous Dirac equation. Most of the deepest equations of physics are also the most beautiful ones e.g. Maxwell's equations of classical electrodynamics, Einstein's equations of general relativity. Beauty is always considered as an important guide in physics. My question is, can/should anyone trust mathematical aesthetics so much that even without experimental verification, one can be fairly confident of its validity? (Like Einstein once believed to have said - when asked what could have been his reaction if experiments showed GR was wrong - Then I would have felt sorry for the dear Lord)
Dear sb1, a good question. Well, beauty is a good guide in physics research but only for those whose sense of beauty is aligned with Nature's sense of beauty. ;-) Dirac was among them, at least when he was writing down his beautiful equation, but many others have a different sense of beauty that can easily lead them off the track.
The right sense of beauty is linked to the equations' rigidity and uniqueness. If a woman is beautiful, you may think that not a single thing could be improved about her. Every correction would damage this beauty. The same thing holds for the beautiful theories and equations in physics that simply "fit together". An important characteristic that can make a theory more rigid and constrained is symmetry - but it is not the only characteristic that can do so. For example, the nontrivial cancellation of various a priori conceivable theoretical problems - such as anomalies - also constrains theories and makes them "prettier" relatively to theories that haven't had to pass any similar theoretical tests.
Why are the equations and theories that "fit together" more likely to be the right description of Nature? Well, unless they're already falsified, they have many fewer parameters waiting to be adjusted than the competing - so far unfalsified - theories that are not so beautiful.
The "posterior" (after the comparison with the reality) probability that the "not so beautiful" candidate theory is valid is, by the Bayesian logic, multiplied by the probability $P(g=g_0)$ that the parameters $g$ take the right values to agree with the reality.
If the overall prior probability for the "beautiful" and "ugly" classes of theories are chosen to be equal, then the "ugly" theory is punished by the extra factor of $P(g=g_0)$, so it becomes less likely that it is the right theory that describes the observations. A more constrained point in the space of theories (constrained by symmetries and special consistency advantages) gets a "higher weight" because it's qualitatively different from the more "generic" or "uglier" points.
Nature has apparently chosen some repeatable laws that apply everything in the Universe (and maybe beyond) and that predict millions of phenomena from a very small amount of information about the laws that has to be known in advance. So it makes sense to extrapolate this observation and assume that the laws of physics are as constrained as possible, and in this sense, they must "fit together" and be "beautiful".
But again, one has to be very careful about this method to look for theories that becomes very unscientific unless the "beauty" of the mathematical structures may be justified by some technical arguments.
There's nothing particularly special about physics as far as beauty goes. And there are plenty of examples of ugly realities in physics:
(1) The values of the CKM mixing matrix are close to unity but not quite. It would be more beautiful if there was some pattern to it. Instead, all we have is the experimental values. Same applies to the MNS mixing matrix, but not such a large degree.
(2) It would be more symmetric if elementary particles did not depend on handedness. In fact, before the discovery of the handedness of the weak interaction it seems to have been widely believe (on the basis of beauty and simplicity) that there was no such dependency.
(3) The masses of the various particles seem quite contrived. No one knows where they come from and their values are all over the map. This is not beautiful. Beauty would have been all the leptons having masses differing by factors of 2, for example.
(4) The renormalization needed in QFT is quite ugly. Nature is beautiful partly because She avoids having to cancel one infinity against another.
(5) While general relativity has a simple and beautiful set of assumptions, the results are ugly even for the simplest case of a non rotating mass, that is, a black hole. Such an object has a singularity at the center. Singularities are certainly not beautiful.
Steven Weinberg has an interesting comment on this subject:
Aesthetically motivated simplicity
Einstein made what from the perspective of today's theoretical physics is a deeper mistake in his dislike of the cosmological constant. In developing general relativity, he had relied not only on a simple physical principle--the principle of the equivalence of gravitation and inertia that he had developed from 1907 to 1911--but also on a sort of Occam's razor, that the equations of the theory should be not only consistent with this principle but also as simple as possible. In itself, the principle of equivalence would allow field equations of almost unlimited complexity. Einstein could have included terms in the equations involving four spacetime derivatives, or six spacetime derivatives, or any even number of spacetime derivatives, but he limited himself to second-order differential equations.
This could have been defended on practical grounds. Dimensional analysis shows that the terms in the field equations involving more than two spacetime derivatives would have to be accompanied by constant factors proportional to positive powers of some length. If this length was anything like the lengths encountered in elementary-particle physics, or even atomic physics, then the effects of these higher derivative terms would be quite negligible at the much larger scales at which all observations of gravitation are made. There is just one modification of Einstein's equations that could have observable effects: the introduction of a term involving no spacetime derivatives at all--that is, a cosmological constant.
But Einstein did not exclude terms with higher derivatives for this or for any other practical reason, but for an aesthetic reason: They were not needed, so why include them? And it was just this aesthetic judgment that led him to regret that he had ever introduced the cosmological constant.
Since Einstein's time, we have learned to distrust this sort of aesthetic criterion. Our experience in elementary-particle physics has taught us that any term in the field equations of physics that is allowed by fundamental principles is likely to be there in the equations. It is like the ant world in T. H. White's The Once and Future King: Everything that is not forbidden is compulsory. Indeed, as far as we have been able to do the calculations, quantum fluctuations by themselves would produce an infinite effective cosmological constant, so that to cancel the infinity there would have to be an infinite "bare" cosmological constant of the opposite sign in the field equations themselves. Occam's razor is a fine tool, but it should be applied to principles, not equations.
(Physics Today, Nov. 2005, p.32)
Although I agree, beauty can be seductive and may be related to our evolutionary development of "pattern forming" in the brain--we seem inclined to find symmetry beautiful. Effective theories are often ugly as sin---take a look at the Standard Model Lagrangian and tell me it is beautiful :)
Download the plain version of the Standard Model Lagrangian Density: [ps][pdf][tex][txt]
Or download a "fun yet soul-crushing exam question" based on it: [ps][pdf][tex][txt]
I was tempted to ask the exam question here to see what response it evoked :)
Beauty in a physical theory is closely related to the beauty of the mathematical formalism used to express it, I think, therefore beauty in physics and mathematics are closely related. For an entertaining and very illuminating account of beauty in math, see this Mathematical Intelligencer article about what makes an equation "beautiful".
Beauty is harmful to theoretical physics if it is canonized. Then you factually deal with a beautiful mathematical physics but not with physics. Remember P. Dirac. He wrote his famous equation to satisfy the unitarity requirement or so. Later on his equation was adopted for a quite different use - QFT with antiparticles and pair creations rather than for description of one electron. Moreover, up to his death P. Dirac insisted that QED he'd created had a wrong interaction (or the total Hamiltonian). He invited researchers to write better Hamiltonians because despite "beauty" of his equation, the physics in QED was implemented in a wrong way. Many remember the UV and IR divergences in this respect. I will tell you: the ugliness (wrongness) of the current QED is easily seen even in the first Born approximation where one obtains the famous elastic cross section, i.e., even before encountering divergences. The first Born approximation does not predict the soft radiation - the process occurring with unity probability. Than means a very bad initial approximation for interacting particles: the field and "particle" subsystems are decoupled in it. No wonder one obtains an "explosion" of the perturbative corrections in the next approximation.