Why did scientists find fractional particles in the experiment of particle collision?

Question

1. Quarks can’t be isolated because of the quark confinement. Then why did physics observe particles with fractional charge coming out in the experiment of deep inelastic scattering?

2. Gluons are the force carrier of the colour force. We found gluons because there was a third jet appeared in the collision of an electron and a positron, but how can we know that was a gluon? Why was that a gluon? The physicist came up with the idea of gluons after or before this experiment? If it was before, how did they came up with it? Wasn’t that hadrons need to be colour neutral so that they attract each other specific enough to explain why they combined each other tightly together?

3. How can gluons interact with each other? Gluons are massless, are they pure energy?

Answer 1

It's important to understand that we don't directly observe quarks and gluons in scattering experiments. What we observe is an unruly shower of well known and understood particles hitting our detectors. To try and understand what happens in the collision we use a mathematical model called the Standard Model. That is, we have equations that describe how quarks and gluons behave, together with parameters such as their masses and interaction strengths, and we use this model to calculate the expected scattering products. If we find that by adjusting the parameters of our model we get a good fit to the data this is evidence (but not proof) that our model is correct.

And we find that the results from our collider experiments are consistent with our model of quarks with fractional charges and massless gluons. Indeed, we have many decades of such experimental results, all of which are consistent with our quark/gluon model, so few physicists doubt that the model is basically correct. That's why you'll see statements like the up quark mass is $2.2$ MeV and the charge $+\tfrac23$. Strictly speaking these results have not been absolutely proven but in practice no-one doubts them.

Now to get to your specific questions:

1. We have never observed particles with fractional charge coming out in the experiment of deep inelastic scattering. We have observed boring old electrons, muons and pions coming out of the deep inelastic scattering experiments and inferred from these that quarks with fractional charges are present, even though confinement ensures we will never see an isolated example.

2. As above we cannot ever see an isolated gluon but the scattering products we can see imply that gluons exist and have the properties attributed to them.

3. Massless particles can interact quite happily. For example, massless photons interact with electrons. Indeed, above the electroweak transition we believe all the elementary particles are massless. Mass is not the fundamental property many believe. Instead it arises from interactions of the elementary particles with the Higgs field.

I should finish by saying that our quark/gluon model is quite possibly an effective model, i.e. a model that works well only as long as the interaction energies don't get too large. If String Theory is correct then neither quarks nor gluons are fundamental but instead are low energy approximations to string states. You'll see lots and lots of speculation about this but there is currently no experimental evidence to confirm or deny it.