I'm studying the following question
P119 On a spherical planet, the refractive index of the atmosphere, as a function of altitude $h$ above the surface, varies according to the formula $$n(h) = \frac{n_0}{1 + \epsilon h},$$ where $n_0$ and $\epsilon$ are constants. Curiously, any laser beam, directed horizontally, but at an arbitrary altitude, follows a trajectory that circles the planet. What is the radius of the planet?
from the book 200 More Puzzling Physics Problems (Gnädig, Honyek, Vigh). The proposed solutions seems a little off to me.
The first approach is by Fermat's principle of least time. The time taken for light to travel around a circular trajectory must not change for a small perturbation in it's path; from there, by equating the flight time for two circular trajectories, we would get $$\frac{2\pi R n_0}{c} = \frac{2\pi (R+h) n(h)}{c} \Rightarrow n_0R=n(h)(R+h)$$ My question for this approach: I was under the impression Fermat's principle applies when considering fixed starting and ending points. That is light travels from point $A$ to $B$ in the least time. In situation like lenses and mirrors, light does take multiple paths going from object to image and we can equate the time taken in all of those, but our situation seems different: Why can we equate the time taken for 2 different circular trajectories if they do not share their starting and ending points? I will add, however, that properly writing the integral, the Euler-Lagrange equations and imposing that $r=\text{constant}$ is a solution, does yield the same condition, although it requires us to assume the perturbation $\delta r$ vanishes at the endpoints; I just don't know why comparing two disconnected paths would work a priori.
The second approach, is to consider the atmosphere to be divided into homogeneous, thin layers. In this model, light suffers total reflection upon arriving at the boundary for the next layer, and travels around the planet in a polygonal path - which tends to a circle as the thickness of the layers tend to $0$.
From the geometric construction above, we equate the sine of the angle of incidence with the sine of the limiting angle - that is: $$\frac{R}{R+h}=\frac{n(R+h)}{n_0}$$
This gives our previous relation. However, the condition for total reflection to occur is that the angle of incidence must be greater or equal than the limiting angle, i.e. $$\frac{R}{R+h}\geq\frac{n(R+h)}{n_0}$$
In other words, the upper layer has to have a sufficiently small refractive index, but there's no intrinsic lower boundary for it's value. As long as the refractive index falls off sufficiently fast with height, this model should work - but again, we do know that equality is required from the previous method.
What am I missing in these two cases?
As an aside, although both approaches are pretty different, I chose not to split this question in 2 as both pertain to the same problem.