Is (rest) mass conserved in special relativity? I don't understand why it is said that the (rest) mass of a system is not conserved in relativity.
I mean, the momentum of a system is conserved (i.e.: it remains constant in a frame of reference without any external influence).
Also the energy of a system is conserved in relativity: it doesn't change without any external agency
and the (rest) mass of a system is just 
$$m^2=E^2-p^2$$
$E$ is constant, $p$ is constant, how can $m$ change?
For example, entropy is not conserved in a system, that means that the entropy of the system will increase spontaneously with time, which is really the case. But, is this true for mass? 
here is the problem from Griffiths and exact solution as given in the Griffiths,
Two lumps of clay, each of rest mass $m$, collide head-on at $3c/5$ and they stick together. Question: what is the rest mass $M$ of the composite lump?
Solution: In this case conservation of momentum is trivial it is zero before and zero after. The energy of each lump prior to the collision is
$$mc^2/\sqrt{1-v^2/c^2} =5mc^2/4.$$ The energy of the composite lump after the collision is $Mc^2$ (since its at rest).
so the conservation of energy says:
$5mc^2/4 + 5mc^2/4 =Mc^2$ and hence
$M=5m/2$.
Notice that this is greater than the sum of the initial masses! Mass was not conserved in this collision; kinetic energy was converted into rest energy, so the rest mass increased.
 A: The rest mass of the system is conserved, it's just that the rest mass of the system isn't the sum of the masses of the parts.
The rest mass of a system is just the length of the total energy-momentum vector. And that vector is conserved, so the length is conserved.
The sum of the rest masses of the parts is not conserved. But that simply isn't the rest mass of the system.
The reason that mass appears to be the sum of the masses of the parts in non relativistic physics is because in non relativistic physics the energy-momentum vectors of the parts point in almost the same direction. That is because the energy-momentum vectors point in the same direction in spacetime as the motion of the particle in spacetime. For vectors pointing in almost the same direction the length of the sum is approximately the sum of the lengths.
A: For the rest mass we have
$$m^2=p^2=p^{\mu}p_{\mu}$$
where $p^{\mu}=(E,\vec{p})$. It is Lorentz invariant, which means the rest mass of the particle is always the same no matter in which frame the observer is in.
While for relativistic mass, it's simply equivalent to the total relativistic energy, which is always conserved.
Note the difference between conservation and invariance.
A: Here is my comment in more details
For any system or single elementary particle mass $M$ is defined as
$$
    M = \sqrt{E^2 - \textbf{P}^2}
$$
where $E$ is total energy and $\textbf{P}$ is total momentum.
For an elementary particle (like electron) mass is always conserved.
For a system $M$ is conserved as long as $EdE - \textbf{P}d\textbf{P} = 0$, in particular mass of a closed system is conserved.
However for a composite system mass of the system is not generally equal to sum of masses of individual constituents (mass is not an "additive" quantity): $M \neq m_1 + m_2+...$.
E.g. consider a system of two non-interacting particles:
$$
    M^2 = (E_1 + E_2)^2 - (\textbf{P}_1 + \textbf{P}_2)^2 = E_1^2 -\textbf{P}_1^2 + E_2^2 - \textbf{P}_2^2 + 2(E_1 E_2 - \textbf{P}_1\textbf{P}_2) = m_1^2 + m_2^2 + 2(E_1 E_2 - \textbf{P}_1\textbf{P}_2)
$$
If $m_1 \neq 0$ you can choose rest-frame of particle 1, where particle 2 moves with some relative speed then $E_1 E_2 - \textbf{P}_1\textbf{P}_2 = m_1 E_2$, and so
$$
    M^2 = m_1^2 + m_2^2 + 2m_1E_2 = (m_1+m_2)^2 + 2m_1(E_2-m_2)
$$
So you see that in this case $M > m_1 + m_2$ because of energy of relative motion $E_2-m_2 > 0$. You can "extract" energy $M - (m_1+m_2)$ from this system.
The most striking example is probably a system of two photons of the same energy $\omega$ moving at angle $\theta$. In this case
$$
    M = 2\omega \sin\frac{\theta}{2}
$$
In case of a bound system of two particles $M < m_1+m_2$ and you need to input at least energy $m_1+m_2-M$ (bound energy) to "break" the system apart.
This can be easily generalised for arbitrary number of particles.
Now if particles were to interact their relative motion and/or composition can change. So $\textbf{P}_1\textbf{P}_2 \neq \textbf{P}_1'\textbf{P}_2'$ and/or $m_1 \neq m_1'$ (e.g. $H$-atom emitted a photon and moved to a lower energy state thus decreasing its mass).
This is why if you simply add up masses before and after the interaction you will have $m_1 + m_2 + ... \neq m_1' + m_2' + ...$. However if the system as a whole remains closed during the interaction $M=M'$.
P.S. I don't consider "relativistic mass" because I find this concept useless and misleading.
A: Yes. The so called rest mass $m_0$ is the magnitude of energy-momentum four vector and is always conserved. It is not just conserved but is also invariant since it is the magnitude of a four vector.
Perhaps you are confused between rest mass and relativistic mass. The relativistic mass is the total Energy divided by $c^2$.
$m\equiv \gamma m_0 $
This relativistic mass is also conserved in all frames but is not invariant like rest mass.
EDIT : After referring to the example you added, I would like to add that conservation of rest mass is a matter of how you define rest mass. If one defines rest mass of the system as mass in the rest frame of the system then the mass will not be conserved in the kind of examples you mention. But if one defines rest mass as the Energy divided by $c^2$ in the frame in which the total momentum is zero then the rest mass will be conserved. You can check that for the example you mention in question.
A: Mass, or more correctly, rest mass is not conserved in special relativity. Particles are able to be created and annihilated in special relativity, for instance, an electron and a positron can interact to produce two photons:
$$e^++e^-\rightarrow 2\gamma $$
Here mass is clearly not conserved, because both the electron and positron are massive but photons are massless. 
In more detail, take the situation where $e^+$ and $e^-$ have opposite momentum, $p_0$. The total energy of the system is then:
$$E_T^2 = 2m_e+2p_0^2$$
where $m_e$ is the electron/positron mass. To conserve energy-momentum in the collision, we require that the photons afterwards have equal and opposite momenta $p_1$, and also that
$$E_T^2 = 2p_1^2$$
which means that
$$p_1 = \sqrt{m_e+p_0^2}.$$
On the other hand, the electron and positron both have mass $m_e$, but the two photons are massless, and hence mass is not conserved.
The error in your reasoning in the question is that the formula
$$m^2 = E^2 - p^2$$
only holds for the energy, mass, and momentum of a single particle, and so does not work when you are talking about the energy and momentum of a system as a whole. 
A: What does conservation of mass mean in classical mechanics?
Weight and mass are the same , we know the mass by weighing it, and if we add 1 kilo of sugar to another kilo of sugar, we will have two kilos of sugar. That is what is meant classically that the mass is conserved. Dissolving a kilo of sugar to a kilo of water will give you two kilos of sirop.
The same is not true at the elementary particle level and where the energies are high enough for special relativity to hold.
In special relativity as  you state ,
$$m^2=E^2-p^2$$
As one of the other answers state, this m^2 is the "length" of the four vector in the four vector space of Lorenz transformations. In a similar way that the length of a ruler is invariant in three dimensions the mass of an elementary particle does not change. But in three dimensions also adding vectors can have a variable result, depending on the angles of the vectors. Similarly in the four dimensions of special relativity the addition of other particles with their energy and momentum  will give an m', but this m' will not be the sum of the constituent particle masses, but the "length" of the new four vector. For this specific system of particles this length will be conserved , if no new four vectors enter the problem. 
The masses though of the elementary particles will add up to less than the invariant mass of the  system they compose, the sum will be the lowest limit(, if all are at rest in the center of mass of the system.) Thus there is no conservation of mass a la classical physics.
