The interesting point in halogen bulbs is that they managge to not, over time, cover the inside of the bulb glass with condensed vapour of the filament (1).
Therefore, the bulb can be smaller, as it's not needed to spread that metal over a large area.
Now, when touching the bulb, a fingerprint is left, which contains various carbon compounds. When switching on the bulb, it carbonizes, leaving slight dark residues (2).
Even if they are barely visible, they cover a significant fraction of the surface area, and make that area somewhat less translucent. That causes a reduction of light output over the livetime of the bulb,
adding up to a significant loss in overall efficiency of the halogen bulb.
On the alternative explanations:
The idea that somehow temperature differences are created, and cause a problem by weakening mechanical stability makes no sense - the kind of glass a halogen bulb is made of, quarz glas, is very stable under heat differences, as it changed it's volume only very little, compared to other glass.
(1) Instead, that vapour reacts with the halogen, to a compound that will subsequently split again into metal and halogen on the surface of the filament.
(2) I assume these carbon residues are long-lasting or permanent because the bulb surface is not hot enough to just burn them off.