Google told me the operation frequency is 125% - 190% of the cutoff frequency. But I did not find any other material for reasoning.
My question: What happens if frequency was too high?
The maximum operating frequency of a waveguide is typically limited by the higher order modes. If you try to use the waveguide at a frequency where multiple modes can propagate, you will get some coupling between these modes, which will cause some signal power to travel faster than the main signal. This generally causes problems depending on how the signal is being used.