How can we experimentally tell the difference between particles with and without rest mass? 
*

*We only observe their decay products and that is what the rest-mass is reconstructed from.

*Also, there is a whole issue of running coupling which means that rest mass per se actually doesn't make sense, it's only theoretical construct and depends on the renormalization scheme (en.wikipedia.org/wiki/Minimal_subtraction_scheme).
I understand that theoretically we have particles with (lepton, fermion) and without(photon, gluon). I wanted to know if somebody can explain experimentally how we can tell the difference from the decay maybe or somehow else?


Question:


*

*How can we tell from experiments that a certain particle, like a quark has rest mass, but a gluon does not? They both leave decay products. So what is specifically in the experiment (I guess deep inelastic scattering) that will tell from the decay product if it shows rest mass or not?

*Or is it that from experiments we can't tell, we just know theoretically the difference between particles with rest mass, and without, and identify them in the experiment, measure the decay's energy, and then say that that was just energy (from a photon, gluon) or rest mass (from a quark, electron, W,Z bozon)?
 A: I'm just going to answer the title question as asked.
There are at least three categories of ways to detect non-zero mass for particles, and each has variations.
The basic methods are

*

*Measure the kinetic energy to momentum ratio ($T/p$).


*Measure the speed. Any value well distinguished from $c$ implies mass.


*Creation or decay kinematics.


*Observe mixing.
Spectrometer-Calorimeter
(Spectrometers measure momentum; calorimeters measure energy)
Relativity makes it clear that the ratio of total energy to momentum of a particle is $R_0 = T/p = E/p = c$ for a massless particle and $R_m = \frac{\gamma-1}{\beta\gamma} c$ for a particle with mass.
Reliably distinguishing that ratio is easy for particles with large mass.
Speed
If a particle is massless it moves at $c$; if massive it move at less than $c$, so any measurement of speed less than $c$ implies non-zero mass. Getting a value for that mass requires a little more work.
Speed can be measured

*

*By time-of-flight, either between two time-resolving detectors or from a known creation time to a single time-resolving detector. For charged particle it is easy to build detectors with nano-second time resolution, so this a straight-forward for particles even as light as electrons.


*For charged particles with a velocity-threshold detector such as a Cerenkov or transition radiation detector.
For neutral particle you have to get them to interact with a charged object in your detector to spot them, which makes this more difficult. None the less, accelerator neutrino speeds can be constrained to be very close to that of light with existing hardware.
Creation/Decay Kinematics
The conservation of four-momentum at a creation or decay vertex means that with good enough information on the motion of all the involved particle and if all the masses but one are known you can find the final mass.
This is easiest to see in a creation context where the annihilation of a particle with its anti-particle
$$ e^+ + e^- \longrightarrow X + \bar{X} \;,$$
can only proceed if the total (center-of-mass) energy is at least twice the mass of species $X$. Actual measurements generally involve the shape of the production cross-section versus total (CoM) energy rather than seeking the actual threshold where the production rate vanishes.
Attempts to obtain the neutrino masses this way have, so far, been frustrated by the difficulty of the experiment and the low rate near the end-point. However, new measurements are contemplated.
Mixing
This is the means by which we know neutrinos must have some mass. In essence, mixing requires time and no proper time passes between points on a luminal trajectory, so anything that mixes can't be following luminal trajectories and therefore must have mass.
A: Theoretically speaking, the simplest indicator is that zero rest mass particles, and only they, travel at the speed of light. But this may be challenging to measure directly. Renormalization is a computational heuristic that extracts theoretical predictions out of QFT (because we only have a perturbative formulation of it), but the parameters that come up in the answers, like effective masses and charges, have to be determined experimentally and inserted into the renormalized expressions by hand. So they are much less "theoretical" than the renormalization itself. There is Einstein's conjecture that matter has no "residual" mass, which never shows up in decays or collisions, needed to derive the mass-energy relation, see Why is Einstein's mass-energy relation usually written as $E=mc^2$, and not $\Delta E=\Delta m c^2$? But even the presence of such mass would not preclude kinematic measurability of the rest mass,
The question is of practical importance for neutrinos, which for a long time were thought to have zero rest mass, because of their role in the structure formation after the Big Bang. The original evidence was indirect, namely that  neutrinos oscillate between flavors during flight. From oscillation  measurements one can determine the mixing angles and hence the  differences between the squares of masses. Finite mass can also be inferred from other effects, like appearance of neutrinos with opposite chirality component in chirality selective experiments, double $\beta$ decay, and some cosmological observations. Kinematic determination of the neutrino rest mass is surveyed in Current Direct Neutrino Mass Experiments by Drexlin et al.  Here is the idea:

"The direct neutrino mass determination is based purely on kinematics without further assumptions. Essentially, the neutrino mass is determined by using the relativistic energy-momentum-relationship $E^2=m^2+p^2$. Therefore it is sensitive to the neutrino mass squared $m^2(\nu)$. In principle  there  are  two  methods: time-of-flight measurements and precision investigations of
  weak decays. The former requires very long baselines and therefore very strong sources, which only cataclysmic astrophysical events like a core-collapse supernova could provide... 
Unfortunately nearby supernova explosions are too rare and seem to be not well enough understood to allow to compete with the laboratory direct neutrino mass experiments. Therefore, aiming for this sensitivity, the investigation of the kinematics of weak decays and more explicitly the investigation of the endpoint region of a $\beta$-decay spectrum (or an electron capture) is still the most sensitive model-independent and direct method to determine the neutrino mass."

A: 
How can we tell from experiments that a certain particle, like a quark, has rest mass, but a gluon does not? They both leave decay products. So what is specifically in the experiment (I guess deep inelastic scattering) that will tell from the decay product if it shows rest mass or not?

At the moment the data from particle physics are fitted well by the standard model and the predictions of the model are validated, but all this is within experimental errors. Experimental errors are the same for the tracks coming out of the interactions, but the interpretation and use of this information make large differences in the accuracy of determination of masses.
Quarks and gluons are dependent on a large number of measured tracks, in jets, and thus an accumulation of errors. Errors are minimized in, for example, in a weak decay of a Z to mu+mu-. Thus the models we use affect the concept of masses for the identified particles. At the moment the standard model particles with zero mass are well validated, so it is an interaction between theory and measurements.

Or is it that from experiments we can't tell, we just know theoretically the difference between particles with rest mass, and without, and identify them in the experiment, measure the decay's energy, and then say that that was just energy (from a photon, gluon) or rest mass (from a quark, electron, W,Z bozon)?

Cart before the horse. The interactions, weak and electromagnetic, first measured zero-mass particles, i.e. in the energy balance of the experimental measurements the mass of the missing particle was within errors zero, and then the theory came to model the data. That is why for so many years we had the neutrinos massless because energy and momentum conservation in particle interactions measured them as massless, within errors.
The mass of the photon being zero is a linchpin not only in the standard model but also in special relativity, which is so well validated that there is not space to doubt that no matter how fine the errors, the mass of the photon is zero. This is not the same for neutrinos. 
It is meaningless to ask for improvements in accuracy with the gluons which are never free to be measured with the strong interactions.
Thus at the moment, we are at the point where we have a standard model theory that encapsulates all our measurements, is predictive and has small windows for physics outside the standard model, which at the moment does not affect the zero masses necessary for the photon and gluon.
Edit after comments:
Here is the generation by a photon  of an e+ e- pair in the field of an electron ( the long line)

The energy-momentum vector of the incoming photon can be fitted by measurements and the mass of the incoming found zero within errors.
Here is a three-jet hadronic event in the ALEPH detector on the mass of the Z.

It is interpreted as a quark-antiquark gluon event, quark-antiquark from baryon conservation number, and gluon because it has baryon number zero. The jets are the results of the hadronization and it is only by fitting a great number of such events by use of the standard model and hadronization models that the conclusion about the masses can be made: the models have the masses that exist in the standard model particle table.
