This phenomenon is known as chromatic aberration, which is the failure of the lens to focus all colors to the same point. The January 2012 20/20 magazine article, Chromatic Aberration: The End of the Rainbow, covers the topic in depth with regards to eyewear.
The chromatic aberration Wikipedia article explains chromatic aberration in general:
Chromatic aberration [...] is a failure of a lens to focus all colors to the same point. It is caused by dispersion: the refractive index of the lens elements varies with the wavelength of light.
The 20/20 article explains that the effects of chromatic aberration are a combination of several factors:
The effects of chromatic aberration result from a combination of the Abbé value of the lens material, the prescription, the point at which the patient’s line-of-sight intersects the lens and the distance from the object being viewed.
The article points out that chromatic aberration will not be noticable when viewed through the center of the lens, but the color displacement may be noticeable otherwise. The extent of color fringing is a function of the Abbe number of the lens, with low Abbe value lenses having color fringes that are further apart.
When the patient’s line-of-sight passes through the optical center, no prism is encountered. When the patient looks away from the optical center, these stacked images are displaced by differing amounts, sort of like spreading a deck of cards across a table. The result is a rainbow of blur or color fringes at the edge of contours within the patient’s field of view. The width of this rainbow is related to the Abbé value and the prism powers for the various wavelengths as determined by Prentice’s Law. The color fringes are further apart from red to blue for a low Abbé lens and closer together for a high Abbé lens. It is important to recognize that a lens has not one power, but differing powers depending on the wavelength that is being considered.
The Abbe number Wikipedia article defines the Abbe number:
In optics and lens design, the Abbe number, also known as the V-number or constringence of a transparent material, is an approximate measure of the material's dispersion (change of refractive index versus wavelength), with high values of V indicating low dispersion. [...]
The Abbe number, VD, of a material is defined as
$$V_D = \frac{ n_D - 1 }{ n_F - n_C },$$
where nshort, ncenter and nlong are the refractive indices of the material at three different wavelengths. The shortest wavelength index is nshort and the longest is *nlong.
The 20/20 article describes the amount of prism encountered when line-of-sight moves away from the optical center:
When the line-of-sight moves away from the optical center, prism is encountered. Prentice’s Law determines the amount of this prism. In the above example, when the line-of-sight moves to 1 cm away from the optical center, the He d line image will be displaced by 10 diopters x 1 cm or 10∆. However the blue portion of the image will be displaced by about 10.16∆ and the image formed by the red portion of the image will be displaced by about 9.84∆. The result is that the image is spread on the retina over a range of about 0.33∆. It is this spread of images that is seen as the rainbow fringes characteristic of chromatic aberration.
The same 10 diopter lens made of a material with an Abbé of 42 will perform differently. Its power for the He d line is 10 diopters, but the power for blue light will be about 10.12 diopters and for red light will be about 9.88 diopters. The dioptric spread in this case is 10 ÷ 42 or about 0.24 diopters rather than the 0.33 dioptric spread with an Abbé of 30. The result is that the fringes are not spread over such a wide distance, and less “blur” is perceived, because a lens with a higher Abbé value was used.
The prism correction Wikipedia article defines prism diopters:
Prism correction is commonly specified in prism dioptres, a unit of angular measurement that is loosely related to the dioptre. Prism dioptres are represented by the Greek symbol delta (Δ) in superscript. A prism of power 1Δ would produce 1 unit of displacement for an object held 100 units from the prism. Thus a prism of 1Δ would produce 1 cm visible displacement at 100 cm, or 1 meter. This can be represented mathematically as:
$$P = 100\,\tan d\!$$
where $P$ is the amount of prism correction in prism dioptres, and $d$ is the angle of deviation of the light.
For a prism with apex angle $a<$ and refractive index $n$,
$$d = (n-1)\,a$$
Finally, the 20/20 article points out that the distance from the object affects the amount of spread:
The location of the point at which the line-of-sight passes through the lens to the optical center of the lens and the power of the lens together determine the “prism spread” of the fringes. If this prism spread is 0.33∆, then it represents a width of 0.33 cm if the object is 1 meter away. If the object is two meters away, the fringes are 0.66 cm wide. At 10 meters away they are 3.3 cm wide. It is apparent that the width of the fringes becomes greater as you look at objects at greater distances.