Can a metallic rectangular waveguide support hybrid modes? Studying the basics of guided waves in metallic rectangular waveguides, filled with homogenous dielectric material, I see that there is always a distinction between TE and TM modes. I understand that a TEM mode is impossible in such a configuration, but I'm wondering if hybrid modes (i.e. modes with both longitudinal components of the electric and the magnetic field nonzero) are possible and allowed by Maxwell's equation.
 A: What really determines the mode structure is the boundary conditions, which in a metal wave guide are particularly simple: the electric field should be zero in metal, that is the component of the electric field parallel to the wall of the wave guide should be zero on the wall. Apart from that the waves are like the electromagnetic wave in vacuum - strictly transversal.
A: In a homogeneous guide for a hybrid TE&TM mode to exist you must have the propagation factors be equal. 
As an example take a rectangular guide of dimensions $a\times b$ then the $TE_{mn}$ and $TM_{mn}$ modes cutoff wavenumber is $\kappa_{m,n} = \sqrt{ \left( \frac{m\pi}{a}\right)^2 + \left( \frac{n\pi}{b}\right)^2}$, and their respective wavenumbers $k_{mn}(\omega)$ are equal to $k_{mn}=\sqrt{\left( \frac{\omega}{c}\right)^2-\kappa_{mn}^2}$.
Still one could not call this a genuine hybrid mode because the amplitude of each mode can be independently chosen from that of the other.
For a circular guide of radius $a$ the $TE$ and $TM$ modes do not even have the same wavenumber. In fact, the cutoff modes have $$\kappa_{mn}(TE)=\frac {p_{mn}'}{r} \\{\kappa_{mn}(TM)=\frac {p_{mn}}{r}}$$ where $J_m'(p_{mn}'a)=0$ and $J_m(p_{mn}a)=0$. For certain combination of indices these may equal, for example $p_{04}'=p_{14}=13.324$, but again you have the same independence of the their amplitudes. So this is not a true hybrid mode.
In general, in a homogeneously filled waveguide the TE and TM modes have independent amplitudes and even if one can select a certain a pair of equal wavenumbers their combination is not a true mode. This is completely analogous to the distinction one can make between a pure state that is the energy eigenfunction of a quantum mechanical system and a linear combination of such states even in the case of a degenerate eigenvalue.
