In space, a photon with momentum $P$ is reflected off a previously at rest mirror, imparting momentum to the mirror. Even if the wavelength of the reflected photon is much longer than the that of the incident, the reflected photon still has some momentum vector pointing the opposite direction, meaning that the momentum of the mirror must be greater than $P$ in order to follow the conservation of momentum of the entire system. But this would all mean that the sum of the magnitudes of the momentum vectors has increased, so somehow the energy of the system has increased? Where did I go wrong?
The wavelength of the reflected photon will be slightly larger, so that its energy and momentum have decreased. So not all of the momentum vector magnitudes increase. The photon loses energy to provide it to the mirror. This effect is tiny, which is why usually we don't notice mirrors moving or wavelengths changing after reflecting off mirrors.
Let's look at this one by one:
I'm assuming here that the mirror does not have infinite mass, and that it is perfectly reflecting (only absorbs energy by the momentum change of the photon, doesn't absorb any energy from the photon directly).
1. Momentum Conservation: We know that the momentum of a photon is p=h/λ. When the photon is reflected, it changes direction. If the mirror was large enough, we could assume that the photon didn't give any energy to the mirror, but this is not the case here.
So, some of the photon's energy (and hence, momentum) goes to the mirror. Let P(1) and P(2) be the initial and final momenta of the photon, P(m) be the final momentum of the mirror, λ(1) and λ(2) be the initial and final wavelength of the photon.
By momentum conservation: P(1)=P(m)+P(2)
Taking direction of P(1) as positive, h/λ(1)=P(m) - h/λ(2) (Since some momentum is lost, P(1)>P(2), and so λ(1)<λ(2). So yes, the momentum of mirror is indeed greater than the initial momentum of photon.
2. Energy Conservation: We know that the energy of a photon is hc/λ. Since we've established that the mirror absorbs some energy (only due to the momentum change of photon), we can say that the photon loses energy, and is redshifted. Again, E(1) and E(2) are the initial and final energies of photon, and E(m) is the final energy of mirror.
By energy conservation: E(1)=E(m)+E(2), and so hc/λ(1)=hcλ(2)+E(m).
So, the mirror does absorb energy(and momentum), but no conservation laws are violated.