Center of mass of two $\gamma$ rays moving in opposite directions Suppose there are two $\gamma$ rays with frequencies $\nu_1$ and $\nu_2$ moving in opposite directions according to a reference frame $S$. I want to find the velocity of the center of mass of this system.
Since photons do not have mass, the center of mass is the frame in which the sums of momenta vanishes.
Let $S'$ be this reference frame. The total momentum in $S'$ is given by:
$$
p'= p_{1}'+p_{2}' = \frac{h}{c}(\nu_{1}'-\nu_{2}')=0
$$ 
Which implies $\nu_{1}'=\nu_{2}'$. 
There frequencies in $S'$ are given by:
$$
\nu'= \left(\frac{1+\beta}{1-\beta}\right)^{1/2}\nu
$$
Therefore, the condition $\nu_{1}'=\nu_{2}'$ gives $\nu_{1}=\nu_{2}$. But $\nu_{1} \neq \nu_{2}$ because the photons can have different frequencies in $S$.
What has gone wrong in the reasoning?
 A: This frame is exist. You got wrong result because you ignored that this two photon move in the opposite direction.
Set that the first photon move along the z axis and the second photon move against z-axis.$\omega_1$ and $\omega_2$ are the frequency of the first and the second photon correspondingly in the reference frame.  In new frame shout be $k'_1=-k'_2$  Let's do the Lorenz transformation for the $k_z$
$$\gamma(\omega_1+\beta\omega_1)=-\gamma(-\omega_2+\beta\omega_2)$$
$$\gamma(\omega_1-\omega_2+\beta(\omega_1+\omega_2))=0$$
Thus we obtain that $\beta=\frac{\omega_2-\omega_1}{\omega_1+\omega_2}$ and $\gamma=\frac{\omega_1+\omega_2}{\sqrt{4\omega_1\omega_2}}$
After that one can check that $\omega'_1=\omega'_2$
$$\omega'_1=\gamma(\omega_1+\beta\omega_1)=\frac{\omega_1+\omega_2}{\sqrt{4\omega_1\omega_2}}(\omega_1+\frac{\omega_2-\omega_1}{\omega_1+\omega_2}\omega_1)=\sqrt{\omega_1\omega_2}$$
$$\omega'_2=\gamma(\omega_2-\beta\omega_2)=\frac{\omega_1+\omega_2}{\sqrt{4\omega_1\omega_2}}(\omega_2-\frac{\omega_2-\omega_1}{\omega_1+\omega_2}\omega_2)=\sqrt{\omega_1\omega_2}$$
A: "Since photons do not have mass, the center of mass is the frame in which the sums of momenta vanishes" this is incorrect, and only valid in the reference frame where the two photons have the same frequency.
To compute the center of mass (if it makes any sense or is useful at all in a relativistic setting), and assuming the two photons are localized, you can use the relativistic mass, $m_{rel}=E/c^2=h\nu/c^2=p/c$, to compute the center of mass in the classical way.
A: In fact, you do not have to find the frame $S'$ because the center of mass is independent of reference frame. It is $\sqrt{(h\nu_1+h\nu_2)^2-(h\nu_1-h\nu_2)^2}=2h\sqrt{\nu_1\nu_2}$.
