Why do different wavelength get impeded more or less when in different materials? Moving with the same speed, but a longer physical distance would imply that the fields oscillate less times in the material, but I don't know why a difference in the number of oscillations would impede the wave- I don't even know why things slow down in general. Why some electromagnetic wave would slow down just because it's entering other electromagnetic fields... It would seem to me that the only factor would be time taken to physically move some electron or something in the direction of the fields... But that seems to simple of an explanation to me.
You have in fact put your finger on the reason for the refractive index change. It is related to moving electrons in the direction of the fields. NB dispersion is a complex phenomenon, so this is necessarily going to be an arm-waving explanation - do not take it too literally!
There is a discussion of the phenomenon in this article. Basically the oscillating electric field of the light wave causes electrons in the medium to oscillate. However these electrons typically have some natural oscillation frequency that does not match the frequency of the light, so we have in effect a driven harmonic oscillator and the phase of the electron oscillations is different from the phase of the light wave. This phase difference is responsible for the refractive index. The oscillating electrons emit light that is out of phase with the original light and which therefore interferes with it. Typically this will slow the light and result in a refractive index greater than one, but near resonant frequencies (e.g. at absorption lines) the refractive index can change rapidly and actually be less than one.
The reason the refractive index changes with the frequency of the light is because as you change the light frequency you are (usually) moving either towards or away from the natural frequency of the electrons and the phase difference changes. Typically you would expect the refractive index near a resonance to look like this:
In another question someone has just cited this question. The answer gives a more mathematical description of the phenomenon. Indeed, had I spotted this question I'd probably have flagged your question as a duplicate if it (although my answer here is more layman friendly! :-).