The phenomenon of reflection is closely related to the phenomenon of absorption, since the reflected intensity depends on the imaginary part of the complex index of refraction, as Feynman explains in The Feynman lectures vol. II chapter 33 and it in turns is related to the absorption coefficient chapter 32. We could then consider that in order of light to be reflected, it should be absorbed and emitted by an atom at the surface of the mirror.
During the absorption and emission, the atom suffers recoil due to conservation of momentum. This recoil manifests as a Doppler shift of light frequency, and it is more noticeable when we are talking about high frequency emission\absorption, like in nuclear phenomena. This is well explained by Mössbauer in his Nobel lecture and this is what turns the Mossbauer effect interesting (the absence of such recoil).
If we neglect the Doppler shift, as you said, we are basically ignoring such recoil. It means that we are considering the photon's momentum negligible compared to atom's momentum, or because the later is too heavy or the first has very low frequency.
In any case, one should not expect to have momentum conservation neglecting the atom recoil. But if you are talking about the Mössbauer effect, the explanation is given by Mössbauer in his lecture
In such a recoilless emission process, the entire excitation energy is transferred to the emitted quantum, and the same holds for the recoilless absorption. Here, the term "recoilless" relates only to the recoil energy transferred in a nuclear transition, and not to the transferred momentum. The value of this transferred momentum is determined by the energy of the gamma quantum and is essentially constant, independent of any change in the internal state of motion of the crystal. This momentum is, therefore, transferred to the lattice in all emission or absorption processes, even in the recoilless processes. It is always absorbed by the crystal as a whole, resulting in a translational velocity that is negligibly small.
So in this case, the mirror velocity changes, but it is negligible.