Why is $SU(3)$ chosen as the gauge group. Why not $U(3)$? Why does it even have to be unitary?
Let me first make a general remark about internal symmetry groups, unrelated to our problem of the correct symmetry group for QCD.
The symmetry must act on Hilbert space as a unitary operator for the conservation of probability.
Now let us turn to the strong interaction. The most important experimental facts were that
- Observed hadron spectrum was understood as that of bound states of quarks.
- The SLAC experiment found that in high energy, deep inelastic scattering, the bound quarks behaved as if they were weakly coupled.
- The measured decay rate of $\pi\to\gamma\gamma$ was nine times greater than expected.
In theoretical language, we thus require the properties of confinement (hadrons) and asymptotic freedom (coupling gets weaker at high energies).
A theoretical result is that only Yang-Mills $SU(N)$ gauge theories exhibit asymptotic freedom.
The question is now, what is the value of $N$? Well, experimental fact number three helps us. If the quarks transform in the fundamental triplet of $SU(3)$, the decay rate is enhanced by $3^2$.