Why does Fermat's principle (optics) not apply to all paths? Feynman's statement of Fermat's Principle regarding optics is the following,

"a ray going in a certain particular path has the property that if we make a small change (say a one percent shift) in the ray in any manner whatever, say in the location at which it comes to the mirror, or the shape of the curve, or anything, there will be no first-order change in the time; there will be only a second-order change in the time. In other words, the principle is that light takes a path such that there are many other paths nearby which take almost exactly the same time." 

Quote taken from http://www.feynmanlectures.caltech.edu/I_26.html
My question is this, doesn't this principle apply to any path, sensible or crazy, between two points? For every possible path, aren't there "many other paths nearby which take almost the same time?"
I can't see why there are limited paths for which Fermat's principle applies, since I can imagine creating many tiny "nearby" variations in any path, leading to a small change in time taken to traverse them.
Why are there limited paths which only have "second-order" changes in time when small variations are applied?
 A: If I have a ball rolling on some type of crazy set of hills and valleys, where is the ball going to want to sit? You might say immediately "at the bottom of a valley, of course!" But let's rephrase that.
We can make an equivalent statement by saying that a ball prefers to settle where, if we were to perturb it slightly, the change in the height (or the potential) is only second order in the perturbation. There are only a few special points where this occurs, and they are maxima/minima/saddle points of whatever crazy potential you've conjurred up. To rephrase this à la Feynman, "A particle in a potential will settle in a position where all neighboring locations have almost exactly the same potential."
This is exactly the same reasoning that Feynman is applying. By saying that the time taken for neighboring paths is almost exactly the same, he means that the variation in the time is at best second order in the variation of the path. This can't happen except for very special paths, just as the ball can't settle just anywhere.
A: In mathematical terms, Feynman's condition that "there will be no first-order changes" is the condition that the first-order (functional/variational) derivatives vanish. For generic paths, this condition is not fulfilled; only for stationary paths. 
