Doubt in getting the different result from invariant quantity in Energy-momentum four vector and energy-momentum conservation We have an antiproton moving with kinetic energy $\frac{2}{3}GeV$ and collides with a stationary proton. The rest mass energy of both are $1GeV$.
I am trying to find the momentum of the antiproton.
For a system $E^2-p^2c^2=(\text{Rest mass energy})^2\tag{1}$
Total energy of antiproton $(E_1)=\frac{2}{3}+1=\frac{5}{3}GeV$
Total energy of proton $(E_2)=1GeV$
Momentum of antiproton is $p_1$ and momentum of proton is $p_2=0$.
$(E_1+E_2)^2-(p_1+p_2)^2c^2=(\text{Rest mass energy of antiproton + Rest mass energy of proton})^2\tag{2}$
$\implies \Big(\frac{5}{3}+1\Big)^2-p_1^2c^2=2^2$
$\implies\frac{64}{9}-p_1^2c^2=4$
$\implies p_1^2c^2=\frac{28}{9}$
$\implies p_1=\frac{\sqrt{28}}{3c}$
But if I use $(1)$ for only the antiproton, then I get
$E_1^2-p_1^2c^2=\text{Rest mass energy of antiproton}\tag{3}$
$\implies \Big(\frac{5}{3}\Big)^2-p_1^2c^2=1$
$\implies \frac{25}{9}-1=p_1^2c^2$
$\implies p_1=\frac{4}{3c}$
The correct answer is $p_1=\frac{4}{3c}$.
I am not able to understand, why I get different result from $(2)$ and $(3)$? Equation $(1)$ is for the system, irrespective of the number of particles in it.
I am very confused why we can't use $(2)$ to find $p_1$?.
Please clarify the doubt.
Addendum
I have forgot to mention that the collision of proton and antiproton results in the formation of two photons.
If I use this information to find momentum of antiproton, then again I get different answer.
As no net external force is acting so energy-mometum remains conserved.
$E_{initial}^2-p_{initial}^2c^2=E_{final}^2-p_{final}^2c^2\tag{4}$
$\implies (E_1+E_2)^2-(p_1+p_2)^2c^2=\Big(\frac{hc}{\lambda_1}+\frac{hc}{\lambda_2}\Big)^2-\Big(\frac{h}{\lambda_1}-\frac{h}{\lambda_2}\Big)^2c^2$
$\implies \Big(\frac{5}{3}+1\Big)^2-p_1^2c^2=0$
$\implies\frac{64}{9}-p_1^2c^2=0$
$\implies p_1=\frac{8}{3c}$
Now I am getting completely different answer.
Why this is so? Why the momentum $p_1$ depends on what is formed after collision?
What is the mistake in this method?
 A: UPDATE2: (See below)
UPDATE: (See below)

In (2), the square of the "rest-mass-energy of the system" is not of the "sum of the rest-masses".

but of the "invariant-mass of the system".
(If it were a totally-inelastic collision, it would be the rest mass of the final particle.)
Without any further information, it would be determined by the left-hand side of the equation.
I think it is very useful to draw an energy-momentum diagram of the system.

Conservation of total 4-momentum implies a polygon of 4-momenta.
(Look at my answer to
Momentum diagram for two colliding Particles , as well as links to other related energy-momentum problems. )
From one the problems mentioned in the link above,
(I used the diagram from Momentum in center of mass-frame out of knowledge kinetic energy in lab-frame
but this involves a much more specialized problem... just focus on the diagram) 
this diagram is similar to your problem, where the target is at rest in this frame.

For your problem, you get an isosceles triangle (in Minkowskian geometry),
formed by the incoming equal-mass particles.

The invariant mass for (2) is the "magnitude of the 4-vector sum of the incoming particles"
(which is not the same as the "sum of the 4-vector magnitudes of the incoming particles").
(You can analyze the problem using trigonometric methods [as you use for a free body diagram], but using hyperbolic trigonometry.)
Try it!

UPDATE:
Based on what you already wrote in (1) and (3), 
one can construct an energy-momentum diagram,
where the 4-momentum vectors are added tip-to-tail.
This represents the sum of the initial 4-momentums.
(The red hyperbola has center at the tip of the first 4 vector.)
That this hyperbola meets the opposite ends of the vectors means that
these two initial 4-momentum vectors have the same magnitude (that is, each of the initial 4-momentums have the same rest mass). [This is the Minkowski analogue of an isosceles triangle!]
The vertical components are the relativistic energies.

The horizontal components are the relativistic momentums.

(Think of adding force vectors on a free-body diagram, or
think of adding a sequence of displacement vectors (following a treasure map).)

The dotted segment represents the resultant vector.
This resultant vector is the total 4-momentum in the system "before",
and must (by conservation) be equal to the total 4-momentum in the system "after".
Find the magnitude of that resultant vector, but use

not the Euclidean distance-invariant using the Pythagorean theorem $\sqrt{E^2+p^2}$

but the special-relativistic Minkowskian invariant $\sqrt{E^2-p^2}$
The magnitude of this resultant vector is the rest-mass of a single particle
if the collision was totally-inelastic. (This is often called the "invariant mass of the system".)  Once you have this resultant vector,
you can find its energy and momentum components [in the frame of this diagram].

If you have final particles (as opposed to a single particle),
then the vector sum of the 4-momentums of those particles have to (by conservation)
add up to that resultant 4-momentum vector.
If those final particles are photons, then the 4-momentums of those photons must be angled at 45 degrees (since each of those photons have its relativistic-energy equal to the magnitude of its relativistic spatial-momentum).
FOR YOU TO DO:

Starting from the origin, can you find two [4-momentum] vectors (with angles +45 degrees and -45 degrees) that vectorially add up to the dotted resultant vector?

Can you find the components of each of those [4-momentum] vectors?

For algebraic simplicity, it is probably better to work with frequencies
rather than wavelengths. If you need wavelengths, do that at the last step.
(This is not that much different from the ordinary problems of adding vectors
of forces or adding vectors of displacements.)

UPDATE2:
For context,
Your previous (4):
\begin{align}
&\implies (E_1+E_2)^2-(p_1+p_2)^2c^2=\frac{h^2c^2}{\lambda_1^2}+\frac{h^2c^2}{\lambda_2^2}-\frac{h^2c^2}{\lambda_1^2}-\frac{h^2c^2}{\lambda_2^2}\\
&\implies \Big(\frac{5}{3}+1\Big)^2-p_1^2c^2=0
\end{align}
Your recent edit so (4) now reads:
\begin{align}
&\implies (E_1+E_2)^2-(p_1+p_2)^2c^2=\Big(\frac{hc}{\lambda_1}+\frac{hc}{\lambda_2}\Big)^2-\Big(\frac{h}{\lambda_1}-\frac{h}{\lambda_2}\Big)^2c^2\\
&\implies \Big(\frac{5}{3}+1\Big)^2-p_1^2c^2=0
\end{align}

*

*In your original version of (4), your right-hand side was
$E_{1f}^2+E_{2f}^2-p_{1f}^2-p_{2f}^2$
(which is incorrect because it is the "sum of the squared-magnitudes" $m_{1f}^2+m_{2f}^2$)... with that error, you did correctly do algebra to get the right-hand side to be zero.

However, it is mathematical fact that the sum of two future-timelike vectors is necessarily future-timelike.. it can't have square-magnitude zero.


*In your revised version of (4), your right-hand side is now physically correct [in accord with my suggestion of expressing the total 4-momentum as the sum of two lightlike  4-momentum vectors for photons in opposite directions].

Symbolically, this is
$(E_{1f}+E_{2f})^2-(p_{1f}^2+p_{2f})^2=(E_{tot,f}^2 - p_{tot,f}^2)$,
which is the square-magnitude of the vector-sum (which is correct).

However, the algebra is incorrect. Expand out your right-hand side... it's not zero. The sum of two lightlike-vectors isn't lightlike unless they are parallel. The right-hand side is necessarily positive
(since the total-4-momentum of the system is timelike, from the initial 4-momentums).



*In general, for a two-particle to two-particle interaction.
\begin{align}
E_{1}+E_{2}&=E_{3}+E_{4}\\
p_{1}+p_{2}&=p_{3}+p_{4}
\end{align}
where
$m_{1}^2=(E_1^2-p_1^2)$, $m_{2}^2=(E_2^2-p_2^2)$,
$m_{3}^2=(E_3^2-p_3^2)$, $m_{4}^2=(E_4^2-p_4^2)$,
and
$m_{SYS}^2=\left((E_1+E_2)^2-(p_1+p_2)^2\right)
=\left((E_3+E_4)^2-(p_3+p_4)^2\right)$.


*If the goal is to find $p_1$, (as I said earlier)
you have enough information from
the 4-momentums of the initial particles.

Intuitively, the initial spatial-momentum of the system in the lab frame
comes from the moving antiproton since the target proton is at rest.


*In my opinion, many problems in relativity are treated only as algebra problems.
However, when treated geometrically and trigonometrically [the reason for many of the algebraic invariants and formulas], there is something more tangible
and can be more easily interpreted physically. (With spacetime and vector methods, it is analogous to working with a free-body diagram for an object in equilibrium.)
