After getting through a course on optics (up to the study of the Michelson interferometer), I'm realizing I have no idea why there is such a thing as diffraction. I can understand why there's refraction and reflection, for instance, and I get there are efficient models for it such as Huyghens', but diffraction seems like magic to me. Could somebody please explain to me the intuitive reason why it exists?
Perhaps consider diffraction as essentially the same effect as refraction at a small scale. How close does the atom or mass actually need to be before it affects the light and you consider the light to be "inside" the medium? The presence of matter near to the light starts to produce the same effect as being inside a medium. Or you could say that in a medium light slows down because it experiences a degree of diffraction. Hope that helps.
To understand diffraction in an intuitive way you should take a look at the historical discovery of the phenomenon. The first accurate report and description of the phenomenon was made by Grimaldi in the year 1665.
He performed a simple experiment similar to the penumbra effect. He projected light from an extended source through a pinhole and measured the output. The corpuscular theory of light propagation, which was accepted at the time, predicted that the shadow behind the screen should be well defined with sharp edges. What Grimaldi noticed was that the transition from light to shadow was gradual rather than abrupt.
The further step was provided by Huygens in 1678. He expressed the intuitive conviction that if each point of a disturbance were considered to be a new sourc of a "secondary" spherical disturbance, then the wavefront at a later instant could be found by constructing the "envelope" of the secondary wavelets. This is the main foundation of the diffraction theory. In essence, the wavelike propagation of light predicts that every point will be a new source of spherical waves.
In 1704, Young strengthened the wave theory of light by introducing the critical concept of interference. The idea states that under proper conditions, light could be added to light and produce darkness.
Then, we have to wait til 1818 when Fresnel combines both theory and states, by making some rather arbitrary assumptions, that the "secondary" wavelets introduced by Huygens might interfere between them. He was able to calculate the distribution of light in diffraction patterns with excellent accuracy. So, in a very simplified way, diffraction is just the "auto" interferences of a wavefront propagating.
Then the rest is just an extend and generalization of that principle through great physicists such as Sommerfeld, Rayleigh, Kirchoff, Maxwell ...
A very good and rigorous mathematical description can be found in the "Introduction to Fourier Optics" from J. W. Goodman.