If one tries to verify the construction of the standard model, one has to find a Lagrangian that is invariant under $U(1)\times SU(2) \times SU(3)$. While it seems kind of logic that the Lagrangian should be invariant under rotation, it seems a little bit arbitrary that it should be invariant under $SU(2)$ and $SU(3)$ transformations. I understand that the $SU(2)$ group is constructed through the Pauli matrices, but still, this alone seems like a poor argument.
2 Answers
As far as we know, within the standard framework of quantum field theory it is arbitrary that the Standard Model has $\mathrm{U}(1)\times \mathrm{SU}(2)\times\mathrm{SU}(3)$ as its gauge group. We choose this group because it predicts the correct particle content and interactions that we observe e.g. in colliders, not because of some compelling theoretical reason to choose this group over any other.
Quantum field theories with many other gauge groups are consistent and perfectly valid theories - they just don't seem to describe our universe (though not all choices see consistent, see "anomalies", "Landau poles", etc.).
We demand our theories to be gauge invariant. By that I mean, we do not pick $G = U(1) \times SU(2) \times SU(3)$ because we "choose so". This pops out due to the bundle nature of the electromagnetic, weak, and nuclear forces. Gauge fields are conveniently described as connections of principal bundles over $\mathbb{R}^{1,3}$. This can be understood by the fact that the path integral and correlation functions of any gauge theory must be invariant by the choice of the fiber coordinate.
Regardless, there is some ambiguity as to the group of the standard model is $G$, we can only locally observe the algebra. There are arguments that $G = U(1)$. There is a lot of info here.