What is so special about calcite that it has double refraction? In this video is explained how the double refraction occurs. But it explains that if the ordinairy ray is horizontal and the extraordinairy is vertical that the first one gets less delayed than the second.
Now that is quite logical for all materials I suppose?! But it looks like that calcite is special because it can double refract. But what is makes calcite structure so special?
min. 3:30
https://www.youtube.com/watch?v=MoZar-gCj3E
 A: Calcite's structure is what makes calcite's structure so special. 
The lattice configuration in this specific material makes it anisotropic. A lot of dielectric materials (most?) are isotropic, or are close enough that we treat them as such, due to the lattice structure being more symmetrical. This symmetry makes the refractive index effectively independent of polarization. Calcite's lattice configuration is not symmetrical, so the electric field interacts with it differently depending on the polarization.
Here is a quick explanation of double refraction resulting from anisotropic media. 
A: The microscopic origin of the double refraction is the lattice arrangement of the crystal. The trigonal system symmetry is what makes calcite structure special compared to non birefringent materials such as glass.
The index of refraction is associated to the electronic polarizability of the atoms. You can think of the electromagnetic wave putting the electronic cloud to oscillate. This originates a delay in the field propagation and we perceive this as a smaller speed of light in the medium. If the density of atoms is different along two different directions, the speed of light and therefore the refraction index is different in those directions.
A: Only isotropic substances (fluids and amorphous materials like glass) and cubic crystals have a simple scalar as the refractive index. Anything else needs two or three indices to describe optical properties (or even more when the material is chiral).
The special thing about calcite is that the difference between its refractive indices is quite large. Calcite is calcium carbonate. In the crystal, all the planar carbonate ions lie in the same orientation. The polarizability is smaller when the electric field is perpendicular to the ion plane than when the electric field is parallel to the plane of ions. This gives a difference in the refractive index.
Lots of other common minerals are birefringent: quartz, corundum, mica, etc etc.
