Why symmetry groups as opposed to e.g. monoid or ring? Why is it that symmetries of a system are usually realized as coming from groups (and categorified versions of it) and not other algebraic structures? Why are the terms "symmetry monoid" or "symmetry ring" not frequent (or existent) in the literature? Wouldn't, for instance, the former allow for a description of irreversible transformations? Is a ring structure for symmetries "too much" to require?
 A: Symmetries form groups because (a) they have an identity ("do nothing"), and (b) each operation is invertible. Non-invertible transformations would form a monoid, but such transformations do not describe a "symmetry" because some property of the system is destroyed by the transformation (you can't get back to the original state -- if you could you would have an inverse).
Rings and algebras do have applications in physics, but because they involve multiple operations they go beyond the simple requirements of a symmetry.
A: When we talk about "symmetries" of some object - we mean that the object "stays the same" under certain transformations. So, let us identify a set of such transformations $G$ and objects $X$. An element of the transformation set $g\in G$ should be able to act on an object $x \in X$. I'll denote this action as function application $g(x) \in X$.
From here, we naturally are able to able to compose these transformations:
$$ g_1,g_2 \in G\quad \Rightarrow \quad g_1g_2 \in G : (g_1g_2)(x) = g_1(g_2(x))$$
This composition is associative and has a unique unit $e \in G : \forall x\in X. e(x) = x$
(Notice the uniqueness of $e$ - important in the following.)
We can get the inverse axiom if we assume that $G$ is finite. In that case for every $g \in G$ there are two powers $a > b > 0$ such that $g^a = g^b$ as a result we've got $g^{a-b} = e$ so we've got the inverse $g^{a-b-1}$. This line of argument should also work for compact manifold structures to get the Lie groups like $SU(2)$.
In my experience, when physicists talk about "symmetry groups" they assume finiteness/compactness without stressing (or, sometimes, being aware of) it.
If we don't assume finiteness/compactness of $G$ then, generally, we don't have the inverse axiom. The most obvious example is fixed-increment translations:
$$T(f(x)) = f(x + 1)$$
Constant functions $f$ are symmetric under $T$, but $T$ and $e$ don't generate a group.
A: Physics uses mathematics as a tool to have theories that describe existing data and can predict new ones. It depends on observations and measurements.
The SU groups came into the study of data slowly. First SU(2), to describe the two baryon states of proton and neutron, that are almost of the same mass. It was useful in nuclear physics studies.
SU(3) first appeared as useful in describing particle data in the eightfold way,

The $Ω^-$ was a prediction , its discovery  it validated the quark model .
The  SU(3) color was found crucial for fitting strong interaction data. And so on.
If in the future experiments and data more complicated mathematical concepts will be needed to get a fit and predict new situations, they will be used.
The choice of the tools is indicated by the data, and not the other way around, at least until/if we have a theory of everything.
