If photon has nonzero rest mass, I have inferred that the gauge invariance is broken and the Maxwell's equations change and Klein–Gordon equation comes into picture. What is its effect in optics such as plane waves etc.?
1 Answer
Yes.. the Proca equations are the Maxwell Equations with photon-mass terms in the Gauss and Ampere laws. Its wave equation in vacuum becomes the Klein Gordon Equation.
One effect is that photons travel slower than c=299,792,458 m/s and that different wavelengths of light travel with different speeds (dispersion) in vacuum.
Quoting from Feynman's Lectures on Gravitation (1962)
(Lecture 2, 2.2 Difficulties of Speculative Theories):
He asked, "Tell me, Professor Feynman, how sure are you that the photon has no rest mass?"
...[snip]...
My answer was that, if we agreed that the mass of the photon was related to the frequency as $\omega=\sqrt{k^2+m^2}$, photons of different wavelengths would travel with different velocities. Then in observing an eclipsing double star, which was sufficiently far away, we would observe the eclipse in blue light and red light at different times. Since nothing like this is observed, we can put an upper limit on the mass, which, if you do the numbers, turns out to be of the order of $10^{-9}$ electron masses.
More effects are described in the following article.
I have included a portion of its table of contents.
"The mass of the photon"
Liang-Cheng Tu, Jun Luo and George T Gillies
Rep. Prog. Phys. 68 (2005) 77–130 ( doi:10.1088/0034-4885/68/1/R02 )
https://iopscience.iop.org/article/10.1088/0034-4885/68/1/R02/
Ch 3 Implications of a photon mass 83
3.1. The dispersion of light 83
3.2. The Yukawa potential in static fields 84
3.3. The longitudinal photon 84
3.4. Special relativity with nonzero photon mass 85
3.5. AB and AC effects with finite photon mass 85
3.6. Monopoles and the photon mass 87
3.7. The Casimir effect for massive photons 88
3.8. Photon mass and blackbody radiation 89
3.9. Other implications