I know that photons don't have mass, but what happens when photons —
even the photons from distant stars — reach us?
Small portion of photons is absorbed by atoms which are part of molecules which are part of retina. The energy of these absorbed photons is transformed to electromagnetic impulse. The produced impulses reach brain unit responsible for image recognition. After it recognized, for example, the star on the sky, the brain (mind) reacts correspondingly.
In your case, when photon reached your eye, you asked stackexchange which caused thousand of views :)
Some other photons are absorbed by body and their energy is transformed to heat (in case is star is Sun).
And lots of photons is dissipated and reflected by body. It means during daylight (star=Sun) or in the deep night (with no electric light) all you see is a reflected light from stars (including moonlight).
Are we merely observing the occurrence of photons moving through space
relative to us, or are we really being "bathed" in photons?
We are really being "bathed" in photons. We are being bathed in visible and invisible spectrum of photons.
I know that when I observe rain, I can both observe it from a distance
but could also be immersed in it as well if in the path of that rain.
But with distant starlight, are we just observing it or are the
photons actually reaching and penetrating the earth around us?
They are reaching and penetrating us and the world around us. Even you see object near you your eyes and brain react when photons reach you and the brain's image recognition unit creates picture for you and illusion that the object is 'there'. So in rain case there are two thing: rain itself (the other story) and light which corresponds to the rain visibility. In latter case it is just star (Sun) light reflected and refracted by water drops.
If they are penetrating, does science tell us what is actually
happening on an atomic or sub-atomic level?
Science tells us the one kind of energy is transformed to another one. Light as electromagnetic kind of energy if transformed to heat when it is absorbed. Or it can be reflected or refracted. On atomic level, the atom changes its energy level to higher or lower if it takes or emits photon. When photon is absorbed it magically disappears and atom becomes hotter...