You have it backward, faster speed of light in a material corresponds to less refraction, not more. In the limit that the speed of light in a material was the same as the speed of light in vacuum, there would be no refraction at all.
It can be shown that in a material the index of refraction is the speed of light in vacuum, $c$ divided by the speed of light in the material $c_m$.
$ n ={ c \over c_m} $
So, slower speed in a material corresponds to a larger index of refraction ,and higher speed to a lower index of refraction. The index of refraction is always greater than or equal to 1, because $c$, the speed of light in vacuum, is always greater than the speed in a material.
So, as you have stated, red light has a lower index of refraction than blue light since it also has a shorter wavelength, so lower index of refraction corresponds to higher speed in a material.
Now Snell's Law is stated
$${n_1 \over n_2} = {\sin \theta_2\over \sin \theta_1}$$
where the geometry is as shown:
So if $n_1=1.0$ and $\theta_1 = 20$ degrees we have
$n_2 =1.51$ for red light and $\theta_2 = 13.09$ degrees,
$n_2 = 1.53$ for blue light, and $\theta_2 = 12.91$ degrees.
As you can see from the diagram, a smaller value of $\theta_2$ corresponds to a larger amount of refraction because it is a larger change in angle and thus a larger change in direction of the incident light.