I can give you two physical explanations.
One crispy:
Well, the photon would disperse in vacuum, it just does not have the time for it. The photon in vacuum travels at the (vacuum) speed of light. This means that whatever happens within the photon's world doesn't get to us, since its time is frozen. If the photon is smiling, you would see it always smiling. Therefore, the width of the photon remains frozen to you, for the whole photon's life, whatever the dynamics that would hold within the photon's world is.
On the other hand, in matter photons are slowed down. This entails that their time gets unfrozen. In turn it means that, whatever happens in the photon's world, leaks out to our world: as a consequence, we see the photon's wavepacket disperse in matter.
One more technical:
A wavepacket, be it related to massive or massless particle, is made of states with different momentum.
Electrons (as well as any massive particle) disperses in vacuum since states with different momentum travel at different speed in vacuum: Some states are slower, some faster, resulting in an overall state getting wider.
Photons, on the other hand, are simple objects in vacuum: Any photon travels at the same speed, irrespectively of its momentum. Therefore all components of the wavepacket travel at the same speed, thus keeping the wavepacket firm, its shape unchanged.
In matter, photons behave like massive particles, exactly because, in matter, photons with different momentum travel at different speed: photon's wavepacket disperses in matter.