It is because the U(2) x U(2)$\rm U(2)\times U(2)$ symmetry breaking is much smaller than the U(3) x U(3) $\rm U(3)\times U(3)$ symmetry breaking, not to speak about U(6) x U(6) $\rm U(6)\times U(6)$ symmetry which is badly violated by large masses of heavy quarks (c,b,t)$(c,b,t)$. The characteristic scale of strong interactions is 1$1$ GeV (the proton mass), while the u,d quark masses are of order a few MeV. Thus the predictions of U(2) x U(2) $\rm U(2)\times U(2)$ symmetry for hadrons must be valid to an accuracy of about 1%$1\%$. The s-quark mass is much larger (around 100 MeV), so U(3) x U(3) $\rm U(3)\times U(3)$ symmetry predictions are not nearly as accurate. Of course, chiral anomalies destroy the axial U(1)$\rm U(1)$ symmetry, so we should really be talking about SU(2) x SU(2) x U(1) $\rm SU(2)\times SU(2)\times U(1)$ instead of U(2) x U(2) $\rm U(2)\times U(2)$ and SU(3)x SU(3)x U(1)$\rm SU(3)\times SU(3)\times U(1)$ instead of U(3) x U(3)$\rm U(3)\times U(3)$.