Can someone please explain the cases that the Effective refractive index $n$ on it can be equal zero? Does such a situation even exist? I am trying to solve an ordinary differential equation to find the dispersion relation for Linearly graded index TM modes but in order to get a solution for the equation $n$ must be equal zero.
There are a few notable cases where permittivity (and hence $n$) can be zero for a specific frequency:
- plasmas have their plasma frequency
- electrons in metals have a plasma frequency, associated with their "plasmon"
- polar optical phonons have their longitudinal oscillation frequency, associated with their "LO phonon"
To be clear, in all of these cases I'm assuming that permittivity has been defined so that all oscillating charges in the material's response are included. In all of these cases, a negative contribution to permittivity arises (some charge displacement is out of phase of the electric field) because of the inertia associated to the charge movements. This negative permittivity contribution just happens to cancel the regular positive background permittivity, for a special frequency.
Since all of these cases have losses, $n$ does not actually go to exactly zero. But, it's quite close, and when we idealize these systems it is sometimes appropriate to neglect the losses and just assume $n$ passes through zero.
In your case, I would advise that when you search for TM modes in whatever structure you have in mind, you should do a calculation including material losses, then allow for attenuation in your mode (either spatial or temporal attenuation, i.e., either complex $k$ or complex $\omega$). This is slightly more complex but it is more realistic, and as a bonus you get information about how good/bad the mode actually is.