You are right to question the assumption that the impulse given to the mirror is twice the incoming photon's momentum, but this is not anything to do with the mirror being a perfect reflector. This is an approximation. You are correct that if the reflected photon's momentum is equal to the incoming photon's momentum, then the mass of the mirror must be infinite (or else, the reflected photon's momentum must be lower). This is another way of saying that the mirror cannot move. The best way to explain is to just do a simple kinematic calculation.
Let's ignore the spring and do a simple elastic collision of mirror and photon. This is a non-relativistic calculation (nothing like Compton scattering), so let's use non-relativistic energy and momentum conservation.
Let's take the incoming photon's wavelength to be $\lambda$, the reflected photon's wavelength to be $-\lambda^\prime$ (negative since it is reflected in direction), and the mass of the mirror as $M$. After the photon reflection, suppose the mirror is imparted a velocity $v$. By momentum conservation,
$$
\frac{h}{\lambda} + \frac{h}{\lambda^\prime} = M v \:,
$$
and by energy conservation,
$$
\frac{hc}{\lambda} - \frac{hc}{\lambda^\prime} = \frac{1}{2}Mv^2\:.
$$
You can convince yourself that after eliminating $h/\lambda^\prime$ and some rearranging you will have an equation that is quadratic in $v$, whose formal solutions will be
$$
v = -c \pm c\sqrt{1+\delta} \:,\:\:\:\:\:\:\textrm{where}\:\:\:\delta = \frac{4 h}{M c \lambda} \: .
$$
I'm skipping a lot of trivial algebra (and the quadratic formula); you should be able to get the result above without too much trouble. We can immediately throw away the unphysical tachyonic solution, and, since $\delta \ll 1$, we can expand in $\delta$ to get
$$
\frac{v}{c} = \frac{1}{2}\delta + O(\delta^2) \:.
$$
Thus, we get
$$
Mv \approx \frac{2h}{\lambda} = 2 p \:,
$$
where we have ignored terms that are higher order in $h/\lambda$ (meaning higher order $\delta$ terms). Thus, the momentum of the mirror is approximately just twice the incoming photon's momentum. In other words, you can just approximate the kinematics of the system as if the reflected photon has the same momentum $p$ as the incoming photon, and that the mirror is therefore given a momentum $2p$ because the photon gets reflected (the photon's impulse must be $-2p$ in order to reverse direction, so the mirror's impulse must be $+2p$ in order to conserve momentum).
In reality, the photon will see some wavelength shift, but it will be small. The leading order term in the mirror's impulse will come from the photon's change of momentum due to reflection. Intuitively, this is because the rest mass energy of the mirror is much larger than the photon's energy. For the sake of intuition, you can pretend the photons are equivalent, here, to small particles of mass $m$, where $m$ is given by $m c^2 = hc/\lambda \ll M c^2$. Think of bouncing a marble on the ground, where the mass of the Earth is much larger than the mass of the marble: each individual photon's momentum will not change in magnitude by much since the mirror's mass is much higher, only its direction will change. This intuition is supported by the analysis above: we would expect our conclusions to break down when $\delta \sim 1$, or in other words, when $h/\lambda \sim Mc$ (ignoring trivial numerical factors).
As an aside, we can also approximate what the wavelength shift will be. The value of $v$ up to first order corrections will be
$$
\frac{v}{c} = \frac{1}{2}\delta - \frac{1}{8}\delta^2 + O(\delta^3) \: .
$$
Thus,
$$
Mv \approx 2 \frac{h}{\lambda} - \frac{2}{Mc} \left(\frac{h}{\lambda}\right)^2 \: .
$$
Placing this expression back into to conservation of momentum equation at the top, we will have
$$
\frac{h}{\lambda^\prime} - \frac{h}{\lambda} \approx - \frac{2}{Mc} \left(\frac{h}{\lambda}\right)^2 \: .
$$
So,
$$
\frac{\Delta p}{p} \approx - \frac{2p}{Mc} \:,
$$
where higher order corrections in $p$ will be suppressed by factors of $1/Mc$. In terms of $\lambda$, this shift will be, up to first order corrections,
$$
\frac{\Delta\lambda}{\lambda} \approx \frac{2h}{Mc\lambda} \:.
$$
So, if we take visible light (say, $\lambda = 5 \times 10^{-7} \,\textrm{m}$), and $M = 0.1 \,\textrm{kg}$, this proportional shift will be about
$$
\frac{\Delta\lambda}{\lambda} \approx 8 \times 10^{-35} \:,
$$
which is absolutely the textbook definition of negligible. Where the mirror picks up its detectable motion is in the sheer number of photons hitting it.
