Why is $SU(3)$ chosen as the gauge group in QCD? Why is $SU(3)$ chosen as the gauge group. Why not $U(3)$? Why does it even have to be unitary?
 A: Let me first make a general remark about internal symmetry groups, unrelated to our problem of the correct symmetry group for QCD.

The symmetry must act on Hilbert space as a unitary operator for the 
  conservation of probability.

Now let us turn to the strong interaction. The most important experimental facts were that


*

*Observed hadron spectrum was understood as that of bound states of quarks.

*The SLAC experiment found that in high energy, deep inelastic scattering, the bound quarks behaved as if they were weakly coupled. 

*The measured decay rate of $\pi\to\gamma\gamma$ was nine times greater than expected.


In theoretical language, we thus require the properties of confinement (hadrons) and asymptotic freedom (coupling gets weaker at high energies). 

A theoretical result is that only Yang-Mills $SU(N)$ gauge
  theories exhibit asymptotic freedom.

The question is now, what is the value of $N$? Well, experimental fact number three helps us. If the quarks transform in the fundamental triplet of $SU(3)$, the decay rate is enhanced by $3^2$.
A: Proving, that $N_c=3$ is not enough to verify, that the transformations are governed by $SU(3)$. Imagine a SU(2) color triplet $\phi \to \phi^\prime=W\phi$.
A meson would be color neutral, because 
$$\phi^\dagger 
\phi \to \phi^{\dagger\prime} \phi^\prime=\phi^{\dagger} W^\dagger W \phi$$
and $W$ is unitary (by definition). 
The problem with that transformation is, that we need to choose a representation of $SU(2)$ acting in the 3D color-space. Here one can choose real $W$s, which leads to a problem. $\phi^\ast$ and $\phi$ transform identically, allowing for color neutral $qq$ or $\bar{q}qq$ particles. Checking, that this is not true for $SU(3)$,
is an exercise in patience and using explicit $\lambda$-matrices. This only leaves the question of Groups $G(N>3)$. As long as it has a subgroup, meeting our criteria, it can be used. But then we just introduce a Group of operations, that we do not need to explain the states we observe. 
