Is the squared norm of the four-momentum of a system of N-particles equal to the total mass of the N particles? I know that for a particle, $p \cdot p=\eta_{\mu\nu } p^{\mu }p^{\nu }=p_{\nu }p^{\nu }=-{E^{2} \ c^{2}+| \mathbf {p} |^{2}=-m^{2}c^{2}}$ is obviously invariant, where $m$ is the rest/proper mass of the particle, $p$ the four-vector momentum and $\eta_{\mu\nu }$ the (Minkowski) metric tensor of special relativity.
What I want to know is, for a system of N particles, each with different momenta, is the sum $(p_1^\mu + p_2^\mu + \dots + p_N^\mu)^2$ equal to the rest mass of the system? It seems to not make sense, since there would be extra cross terms and surely $((p_1^\mu)^2 + (p_2^\mu)^2 + \dots + (p_N^\mu)^2)$ should be the total rest mass? Is $(p_1^\mu + p_2^\mu + \dots + p_N^\mu)^2$ instead equal to just the TOTAL mass of that system? My thinking is that $(p_1^\mu + p_2^\mu + \dots + p_N^\mu)^2 = -{E_{tot}^{2} / c^{2}+| \mathbf {p_{tot}} |^{2}=-M_{tot}^{2}c^{2}}$ where $M_{tot}$ is (maybe) the relativistic mass. If this is true, what does $M_{tot}$ even mean? I thought the total mass of a system would be the sum of the rest masses, unless of course it's talking about relativistic mass which is dependent on the motion.
Could someone please shed light as to why it does kind of make sense, but I would like a more rigorous explanation? Thanks.
 A: In special relativity, rest mass of a system is proportional to net rest energy of that system:
$$
m = \frac{E}{c^2}.
$$
If in systems' rest frame system components move faster they have more energy, hence the system has higher rest energy and corresponding rest mass.
One cannot just add rest masses of the components, that would ignore their kinetic energy. And in more complicated situations where the components interact, it would ignore the potential energy of interaction.
A: Wikipedia: invariant mass
The quantity you are asking about is known as invariant mass. You correctly point out that it only makes sense as the sum of rest masses (up to factors of $c^2$) in non-relativistic limit. If particles are moving relativistically it loses a simple interpretation as the sum of rest masses but it is still a useful Lorentz-invariant quantity. As pointed out in the other answer, it is equal to rest energy, i.e. energy in the frame where system as a whole does not move (which means its' spatial momentum is zero):
$$p_1^\mu + p_2^\mu + ... + p_N^\mu = P^\mu = (E/c,0,0,0)\Longrightarrow (p_1^\mu + p_2^\mu + ... + p_N^\mu)^2 = M_{inv}^2c^2 = \dfrac{E^2}{c^2}$$
If net spatial momentum is non-zero, then from this definition the net energy and net momentum are connected to each other by exactly the same formula as for one particle:
$$p_1^\mu + p_2^\mu + ... + p_N^\mu = P^\mu = (E'/c,\mathbf{P'})\Longrightarrow (p_1^\mu + p_2^\mu + ... + p_N^\mu)^2 = M_{inv}^2c^2 = \dfrac{E'^2}{c^2} - \mathbf{P'}^2$$
