When calculating (elastic) collisions, both energy and momentum conservation is taken into account. The photon initially has the momentum $p_{\gamma i} = \hbar k$ and the energy $E_{\gamma i} = \hbar \omega$, with $k$ being the wave number and $\omega$ the frequency, and the mirror has the momentum $p_{mi}=0$ and the Energy $E_{mi} = mv^2/2 = 0$. With $i$ denoting initial and $f$ denoting final quantities, energy and momentum conservation are
$$
p_{\gamma i} + p_{mi} = p_{\gamma f} + p_{mf} \quad \Leftrightarrow \quad \hbar k_i = -\hbar k_f + m v_f \quad \Leftrightarrow \quad \frac \hbar c \omega_i = - \frac \hbar c \omega_f + m v_f~, \tag{1}
$$
$$
E_{\gamma i} + E_{mi} = E_{\gamma f} + E_{mf} \quad \Leftrightarrow \quad \hbar \omega_i = \hbar \omega_f + \frac 12 m v_f^2 \quad \Rightarrow \quad \omega_f = \omega_i - \frac{m}{2\hbar} v_f^2~.
$$
The negative sign in $p_{\gamma f} = -\hbar k_f$ is needed, because the photon propagates in the opposite direction, so the wave vector points in the opposite direction, too.
Combining the above equations, there follows
$$
\frac \hbar c \omega_i = - \frac \hbar c \omega_i + \frac{m}{2c} v_f^2 + m v_f \quad \Leftrightarrow \quad v_f^2 + 2 c v_f - \frac{4\hbar}{m} \omega_i = 0 \quad \Leftrightarrow \quad (v_f + c)^2 - c^2 - \frac{4\hbar}{m} \omega_i = 0~.
$$
$$
\Rightarrow v_f = \sqrt{c^2 + \frac{4\hbar}{m}\omega_i} - c~. \tag{2}
$$
And using (1) yields
$$
\omega_f = -\omega_i + \frac{cm}{\hbar} v_f~.
$$
This means, $|\omega_f| < |\omega_i|$, so the photon looses the energy which the mirror gets and energy conservation is not violated.
Edit:
The last paragraph of the original post is not wrong, but I got a little confused there and, since $c/\hbar$ is very large, it does not really help in pointing out, why $|\omega_f| < |\omega_i|$. So, let's try again:
It holds
$$
\frac{cm}{\hbar} v_f \overset{\text{(2)}}= \frac{cm}{\hbar} \left( \sqrt{c^2 + 4 \frac{\hbar \omega_i}{m}} - c \right) = \frac{c^2 m}{\hbar} \left( \sqrt{1 + 4 \frac{\hbar \omega_i}{c^2 m}} - 1 \right)~.
$$
The Taylor-expansion of $\sqrt{1+4\epsilon}$ at $\epsilon = 0$ is
$$
\sqrt{1 + 4 \epsilon} = \sqrt 1 + \left[ \frac{4}{2 \sqrt{1 + 4 \epsilon}} \right]_{\epsilon =0} \epsilon + \frac 12 \left[ - \frac{16}{4 \sqrt{1 + 4 \epsilon}^3} \right]_{\epsilon =0} \epsilon^2 + \mathcal O(\epsilon^3) = 1 + 2 \epsilon - 2 \epsilon^2 + \mathcal O(\epsilon^3)~.
$$
Setting $\epsilon = \hbar \omega_i / (c^2 m)$ (which is very small for every reasonable $\omega_i$ and $m$), we get
$$
\omega_f = - \omega_i + \frac{cm}{\hbar} v_f = - \omega_i + \frac{c^2 m}{\hbar} \left( \sqrt{1 + 4 \epsilon} - 1 \right) \\= - \omega_i + \frac{c^2m}{\hbar} \left( 1 + 2 \frac{\hbar \omega_i}{c^2m} - 2 \left(\frac{\hbar \omega_i}{c^2 m} \right)^2 + \mathcal O(\epsilon^3) -1 \right) \\= -\omega_i + 2\omega_i - \underbrace{2 \frac{\hbar\omega_i^2}{c^2 m}}_{\text{very small}} + \mathcal O\left( \left( \frac{\hbar \omega_i}{c^2 m} \right)^3 \right)~.
$$
$$
\Rightarrow \omega_f = \omega_i - \underbrace{2 \frac{\hbar\omega_i^2}{c^2 m}}_{\text{very small}}+ \mathcal O\left( \left( \frac{\hbar \omega_i}{c^2 m} \right)^3 \right)~.
$$
