2 protons collision (both with different kinetic energies) - I don't know what to put in for $p^2c^2$ The problem statement: 

Two protons with kinetic energies $W_{k1}=4GeV$ and $W_{k2}=2GeV$
  colide and form new particles. What is the mass of newly born
  particles? There are as many as possible new particles.

Relevant equations here are:
\begin{align}
E_{before}&=E_{after}\\
p_{before}&=p_{after}\\
E^2 &= {E_0}^2 + p^2c^2 \longleftarrow \substack{\text{Lorentz invariant}}
\end{align}
First I wrote the energy conservation law: 
\begin{align}
E_{before} &= E_{after}\\
E_{k1} + E_{k2} + 2E_{0p} &= \sqrt{{E_{0~after}}^2 +p^2c^2}\longleftarrow\substack{\text{Here the $E_{0~after}$ is a full}\\\text{rest energy after colision}}\\
E_{k1} + E_{k2} + 2E_{0p} &= \sqrt{\smash{\underbrace{\left( 2E_{0p} + E_{0m} \right)^2}_{\substack{\text{after collision we have}\\ \text{2 $p^+$ and new mass $m$}}}} +p^2c^2}
\\
\\
\\
\\
\\
\end{align}
At this point I am not sure what to do with the last part $p^2c^2$. The only thing I came up with was to set:
\begin{align}
pc &= \sqrt{{E_{k~before}}^2 - 2E_{k~before}E_{0~before}}\\
pc &= \sqrt{\left(E_{k1} + E_{k2}\right)^2 - 2\left(E_{k1}+E_{k2}\right)2E_{0p}}\\
\end{align}
If I continued with this calculation I got $W_{0m}=1.87GeV$ while in the solutions it is said to be $W_{0m}=5.634GeV$. Where did I go wrong?
 A: Presumably what you want to do is think in the center-of-momentum frame. In this frame, there is no net momentum, just energy. So presumably you want to convert to a system of particles at rest, two of which are protons and the rest of which are unspecified. How much energy is left over to go into the unspecified other stuff?
Although I think it's most convenient to think about this in the center-of-momentum frame, you can also do the calculation in another frame.
I don't really understand your notation, and it's kind of odd to talk about "kinetic energy" for a relativistic particle, but presumably it means the total energy $E = mc^2 + W$ where $W$ seems to be your notation for "kinetic energy." In that case, what are the momenta of the two particles? What is the total momentum in the frame you're working in? The total energy? The invariant mass?
A: As I said in the comments it really depends on what happens. Here I will assume the protons are colliding heads-on and after collision, all the particles will move with the same speed(so the square of their total momentum will be minimum).
$$E_1=\sqrt{m_p^2c^4+p_1^2 c^2} \\ E_2=\sqrt{m_p^2c^4+p_2^2 c^2} \\ \Rightarrow E=E_1+E_2$$
After collision, as I stated the momentum will be $p_1-p_2$(heads-on collision), so we will have:
$$E^2- \left(p_1 -p_2\right)^2c^2=m_{after}^2c^2=(2m_p+m)^2c^4 \\ \Rightarrow mc^2= \sqrt{E^2-(p_1c-p_2c)^2}-2m_p c^2$$
which I calculated to be $4.1 \text{GeV}$. I think this shows the spirit of solutions to these type of questions, although its details depends on what is exactly stated in the problem.
