Is the rest mass of a system invariant even if it is a function of time? If we imagine a system of particles, and we consider only $N$ of them, then the rest mass of these $N$ particles is given by
$$\left(\sum_{i=1}^N P_i\right)^2=P_T^2=m^2$$
Where $P_i$ is the 4-momentum of the $i^{th}$ particle.
Now, if these $N$ particles are allowed to interact with the other particles in the system, the energy and momentum of the $N$ particles will change with time and therefore we can expect $m=m(t)$. Since different frames disagree on time and the temporal order of events, it seems reasonable to assume that $m(t)$ will be different in different frames and therefore the rest mass of the system is not invariant under Lorentz transforms.
However, $P_T$ is a valid 4-vector therefore its norm should indeed be a Lorentz invariant.
How do we reconcile these two understandings?
 A: Let us suppose  to deal with a finite number of material points in Minkowski spacetime. Assume that
(a)  the interactions in your set of material points are localized  at single (generally multiple) events, say vertices,  in spacetime, where many wordlines meet (the number of incoming lines may be different of the number of outgoing lines);
(b) the world lines are causal future-oriented godesics outside the vertices;
(c) at each vertex the entering total 4-momentum equals the exiting total 4-momentum,
then
the total 4-momentum $P^\mu$ computed just by adding the 4-momenta localized at different places on a flat spacelike section crossing the worldlines does not depend on the section (the adopted inertial reference frame) and also on time. In this case there is a notion of total 4-momentum and total mass $M^2 = -P_\mu P^\mu$ independent of the inertial reference frame.
Notice that we can add vectors applied to different points in Minkowski spacetime because that spacetime is an affine space
The problems arise in curved spacetime or when, in Minkowski spacetime,  the interactions are not localized, i.e., the worldlines outside the vertices are not geodesics. In this case, in Minkowki spacetime,  these interactions have to be taken into account with a suitable stress energy tensor $T^{\mu\nu}$ -- which includes both the material points and the interactions -- and a constant total invariant mass $M$ can be defined provided the fields describing the interactions vanish sufficiently fast at spatial infinity. In this case, $$P^\mu := \int_\Sigma T^{\mu\nu} n_\nu d\nu_\Sigma$$ and $$M^2 = -P_\mu P^\mu\:.$$
In this case the mass is not just the sum of the masses of the points but it also includes a contribution of the interactions.
A: You've stumbled upon a subtle feature of relativistic dynamics that is rarely mentioned in books. The sum of four-momenta of particles at different points is not a four-vector. The reason is precisely the issue you mentioned: the Lorentz transformation effect depends on the spacetime point. By summing four-momenta at the same time in a given frame, you get an extra thing that has to transform between frames.
To make this more precise, we should start with the stress-energy tensor $T^{\mu\nu}$, which is conserved, $\partial_\mu T^{\mu\nu} = 0$. It can be shown that if you integrate a conserved rank $n$ tensor field over space, then you indeed get a rank $n-1$ tensor, which in this case is the total four-momentum,
$$P^\mu = \int T^{\mu 0} \, d^3x.$$
However, the derivation only works when you integrate over all of space. If you integrate over only a subset, then you get extra boundary terms that screw up the transformation properties, so the four-momentum of a subset of a system does not necessarily transform as a four-vector.
The reason this subtlety isn't mentioned in introductory books is because they largely deal with trivial problems in relativistic dynamics, such as collisions where everything happens at one point, and all particles have constant four-momentum before and after; in this case the effect you mentioned doesn't matter. But there are lots of relativistic paradoxes where it does, such as this famous capacitor paradox, which is discussed on this site here.
