To put it briefly, in the classic thought experiment of a massless box with mirrored insides containing photons, does the relativistic mass of the photons imbue the box with rest mass?
I take it that's the case, because I think that's how baryons are supposed to get their mass, but I'm not really getting how this is happening exactly.
The rest mass arises from a difference in the photon pressure against different walls of the box. For example, when the box is stationary in a gravitational field, photon pressure on the bottom of the box is higher than it is on the top, because photons are altered by the field. For another example, when the box is being accelerated by an external force, inertia arises from the fact that the photon pressure on the front wall is less than the pressure on the back wall.
You can only say that if the components are non-interacting.
As soon as you introduce interactions between the components the mass of the system is neither the sum of the rest masses of the components nor the sum of the "relativistic masses".
I've done a moderately thorough treatment in a previous answer.
Relativistic mass of components gives system rest mass?
The concept of relativistic mass, useful when discussing spaceships at the velocity of light, because it is the instantaneous Newtonian inertial mass, is not used for particle physics, as confusing. It is not a conserved quantity.
Photons are elementary particles, and are mathematically described by their four momentum , by four vectors $(E,p_x,p_y,p_z)$ . The "length" of this four vector characterizes uniquely the particle because it is invariant under Lorentz transformations by construction, and it is called invariant mass for that reason.
The invariant mass is the energy of the particle when the momentum is zero, at rest.
The photon has zero invariant mass and it is never at rest always moving with the velocity of light.
Two photons have an invariant mass, unless they are collinear: The $π^0$ meson of mass 135 MeV decays into two photons. The system of these two photons has a four vector with invariant mass 135 MeV.
Thus the addition of a number of photons will have an invariant mass. This summed invariant mass will react as an inertial mass macroscopically.
I think that's how baryons are supposed to get their mass
The quarks within hadrons have very small invariant masses, it is the summed four momenta of all the components,( valence quarks, sea of quark antiquark gluon) within a hadron that gives the final mass of the hadron. See this link for the complicated model.
