Relativistic mass of components gives system rest mass?
The concept of relativistic mass, useful when discussing spaceships at the velocity of light, because it is the instantaneous Newtonian inertial mass, is not used for particle physics, as confusing. It is not a conserved quantity.
Photons are elementary particles, and are mathematically described by their four momentum , by four vectors $(E,p_x,p_y,p_z)$ . The "length" of this four vector characterizes uniquely the particle because it is invariant under Lorentz transformations by construction, and it is called invariant mass for that reason.
The invariant mass is the energy of the particle when the momentum is zero, at rest.
The photon has zero invariant mass and it is never at rest always moving with the velocity of light.
Two photons have an invariant mass, unless they are collinear: The $π^0$ meson of mass 135 MeV decays into two photons. The system of these two photons has a four vector with invariant mass 135 MeV.
Thus the addition of a number of photons will have an invariant mass. This summed invariant mass will react as an inertial mass macroscopically.
I think that's how baryons are supposed to get their mass
The quarks within hadrons have very small invariant masses, it is the summed four momenta of all the components,( valence quarks, sea of quark antiquark gluon) within a hadron that gives the final mass of the hadron. See this link for the complicated model.