How can we tell from experiments that a certain particle, like a quark, has rest mass, but a gluon does not? They both leave decay products. So what is specifically in the experiment (I guess deep inelastic scattering) that will tell from the decay product if it shows rest mass or not?
At the moment the data from particle physics are fitted well by the standard model and the predictions of the model are validated, but all this is within experimental errors. Experimental errors are the same for the tracks coming out of the interactions, but the interpretation and use of this information make large differences in the accuracy of determination of masses.
Quarks and gluons are dependent on a large number of measured tracks, in jets, and thus an accumulation of errors. Errors are minimized in, for example, in a weak decay of a Z to mu+mu-. Thus the models we use affect the concept of masses for the identified particles. At the moment the standard model particles with zero mass are well validated, so it is an interaction between theory and measurements.
Or is it that from experiments we can't tell, we just know theoretically the difference between particles with rest mass, and without, and identify them in the experiment, measure the decay's energy, and then say that that was just energy (from a photon, gluon) or rest mass (from a quark, electron, W,Z bozon)?
Cart before the horse. The interactions, weak and electromagnetic, first measured zero-mass particles, i.e. in the energy balance of the experimental measurements the mass of the missing particle was within errors zero, and then the theory came to model the data. That is why for so many years we had the neutrinos massless because energy and momentum conservation in particle interactions measured them as massless, within errors.
The mass of the photon being zero is a linchpin not only in the standard model but also in special relativity, which is so well validated that there is not space to doubt that no matter how fine the errors, the mass of the photon is zero. This is not the same for neutrinos.
It is meaningless to ask for improvements in accuracy with the gluons which are never free to be measured with the strong interactions.
Thus at the moment, we are at the point where we have a standard model theory that encapsulates all our measurements, is predictive and has small windows for physics outside the standard model, which at the moment does not affect the zero masses necessary for the photon and gluon.
Edit after comments:
Here is the generation by a photon of an e+ e- pair in the field of an electron ( the long line)
The energy-momentum vector of the incoming photon can be fitted by measurements and the mass of the incoming found zero within errors.
Here is a three-jet hadronic event in the ALEPH detector on the mass of the Z.
It is interpreted as a quark-antiquark gluon event, quark-antiquark from baryon conservation number, and gluon because it has baryon number zero. The jets are the results of the hadronization and it is only by fitting a great number of such events by use of the standard model and hadronization models that the conclusion about the masses can be made: the models have the masses that exist in the standard model particle table.