This is explained well in Feynman's lectures on physics
You can think of the light as a wave that causes the electrons in the material to oscillate, much like an harmonic oscillator with friction. There are certain resonance frequencies, frequencies where the oscillators respond more powerfully to the wave. The first significant frequency in transparent materials would often be in the ultraviolet.
Now, red light has low frequency. As you approach the first resonance frequency from low to high frequency (corresponding to red to violet light), you will find that you interact more and more with the material (the oscillations increase) and hence the index of refraction will go higher.
Now, things get more complicated when you get very close to above the resonance frequency, as then you can get so-called "imaginary" parts to the index of refraction and absorption, but that's the general idea.