For low voltage Zeners, a quantum tunneling effect is responsible (https://en.wikipedia.org/wiki/Zener_effect). In this case you can understand the sharpness of the Zener knee in terms of how quantum tunneling tends to be very sharply dependent on barrier strength. Though, if you look at a datasheet you'll see that the Zener knee isn't sharp enough (in my opinion) with these low voltage Zeners.
Microscopically you can imagine that the electrons in the valence band, coming from the p-type side (moving from left to right, beneath the red line, in the figure), ram into the depletion region and bounce off it as a sort of barrier, since there is an interval of positions with no states at their energy. Classically they would all just bounce back, however quantum tunneling means that their wave function can reach a bit into the depletion region. If the lateral distance between valence and conducton bands is made small, then they can tunnel into the conduction band. Rather than going "up" into the conduction band, the electron goes "sideways" into the conduction band. Note this jump happens keeping the total energy of the electron constant: it gives up some of its electric potential energy to pay for the interband transition.
For large voltage Zeners (>5 V or so) there are avalanching effects as well, that multiply the tunnel and leakage currents. This makes the Zener knee even sharper, however it also multiplies the shot noise. Microscopically this process is quite different from what I explained above.
The figure above comes from https://commons.wikimedia.org/wiki/File:Quasi-Fermi_levels.png .