Why $SU(3)$ and not $U(3)$?

Is there a good reason not to pick $U(3)$ as the colour group? Is there any experiment or intrinsic reason that would ruled out $U(3)$ as colour group instead?

Suppose that $\text{U}(3)$ was the gauge group. We can decompose this as $$\text{U}(3)=\text{U}(1)\times\text{SU}(3),$$
which implies that in addition to the $\text{SU}(3)$ that has eight generators corresponding to eight gluons, there would be an additional generator for $\text{U}(1)$. The latter in principle corresponds to an additional gauge boson, but a theory of the strong interactions containing such a particle is inconsistent with experiment.