Vermiculite is a type of clay. Clays in general form structures composed of sheets made from two layers of tetrahedrally bonded silica with octahedrally co-ordinated metal ions in the middle. See http://en.wikipedia.org/wiki/Montmorillonite for a typical structure (montmorillonite is an archetypal clay much beloved of colloid scientists).
Anyhow, in clays the sheets are loosely bound to each other and easily separated. For example in montmorillonite this can be achieved just be suitable ion exchange, and you get a clear gel of separated sheets in water. Vermiculite doesn't do this, but if you heat the solid the sheets will separate to form a fluffy expanded solid with lots of entrained air. The resulting material is an excellent insulator because it contains so much trapped air. It's not the best insulator but it's particularly good in construction because it's uninflammable so it presents no fire risks.
So it's not the expansion at 870C that gives vermiculite it's high R value, it because it has been heated to 870C (then cooled again) to give the fluffy expanded structure.
The expansion of the vermiculite is called exfoliation, and it's normally attributed to vaporisation of water trapped between the silicate layers. I'm not sure how much fundamental research has been done on the mechanism.