You'll see an image (or piece of an image) of the Sun anywhere the surface of the water is oriented such that the angle between the Sun and the water and the angle between the water and your eye are equal:
For water with many ripples, there will be many such locations, but most will be along the line between you and the Sun, or close to it.
In practice, the surface is never perfectly calm, so the effect from ripples always comes into play at least a bit.
I can think of another possible effect that could contribute, even if the water were perfectly calm: on its way in from the Sun, light goes through the atmosphere, which has varying density/temperature, and so also varying refractive index. Now the situation isn't quite so simple as a bunch of parallel rays reflecting off of a mirror. Curved paths through the atmosphere mean that the incident rays are not necessarily parallel, so reflections from different locations (with different incident angles) can meet at a point (the observer). I'd guess that this would be a smaller effect than the ripples, except perhaps for some very contrived weather setups. The effect would still be spreading out mostly along a line between the Sun and observer, since the atmosphere is for the most part varying in the vertical direction (so refraction in a vertical plane).