A set of light rays emitted from a point of an orange hits a smooth and flat surface. My question is: The angles of incidence are different, light rays are reflected towards different directions, why is a distorted image not formed? Why is this not an example of diffuse reflection?
That's precisely why a single sharp image is formed. Extend all three reflected rays back below the surface to see where the light seems to come from. If your drawing is precise enough, you'll find that they intersect at a single point, which is precisely the mirror image of the object.
The point of specular reflection is that the light seems to come from the mirror image. You already get divergent rays from each point even from the original object, and the mirror image does the same. You'll only get parallel rays, if the incident rays were parallel (e.g. from an object that is effectively infinitely far away).