I don't understand why it is said that the (rest) mass of a system is not conserved in relativity. I mean, the momentum of a system is conserved (i.e.: it remains constant in a frame of reference without any external influence). Also the energy of a system is conserved in relativity: it doesn't change without any external agency
and the (rest) mass of a system is just $$m^2=E^2-p^2$$
$E$ is constant, $p$ is constant, how can $m$ change?
For example, entropy is not conserved in a system, that means that the entropy of the system will increase spontaneously with time, which is really the case. But, is this true for mass?
here is the problem from Griffiths and exact solution as given in the Griffiths,
Two lumps of clay, each of rest mass $m$, collide head-on at $3c/5$ and they stick together. Question: what is the rest mass $M$ of the composite lump?
Solution: In this case conservation of momentum is trivial it is zero before and zero after. The energy of each lump prior to the collision is $$mc^2/\sqrt{1-v^2/c^2} =5mc^2/4.$$ The energy of the composite lump after the collision is $Mc^2$ (since its at rest). so the conservation of energy says: $5mc^2/4 + 5mc^2/4 =Mc^2$ and hence $M=5m/2$.
Notice that this is greater than the sum of the initial masses! Mass was not conserved in this collision; kinetic energy was converted into rest energy, so the rest mass increased.