Why should symmetries be described by invertible transformations? Context : I am not sure if this question belongs more in Math SE or here. In fact, my question is similar to that one, but I am dissatisfied with the answers. I think physicists perspective could be interesting.
Also, the question is not about symmetries in quantum mechanics or classical mechanics, but about the general concept of symmetry (for example, when we talk symmetries of the square or when we say that the human body has a plane of symmetry). Examples from physics are welcome however.
First question : In my intuition we can translate symmetries in mathematical language by treating them as functions or maps acting on sets. Is this intuition correct, and how to make it clearer? What would these sets be associated with?
Second question : Given that the above intuition is correct, we can think about the properties of the composition of symmetry operations. The fact that the identity function will always be a symmetry is obvious. Associativity of symmetry operations is also obvious since function composition is associative. But I can't get my head around the fact that the existence of an inverse should be a necessary property of a symmetry operation. Monoids are algebraic structures reassembling groups but lacking this inverse property. Why could not they be used to represent symmetries?
 A: First question
Yes, symmetries can be thought of as maps on a set, but more generally one may study the concept of a symmetry on its own, without relation to the object it acts on.
More precisely, the mathematical structure of a symmetry is a group. Which satisfies a bunch of properties which I'm sure you are already familiar with. But then the space of invertible functions on a set is also a group! So you can "represent" your abstract group element $g\in G$ with a function $f_g:X\to X$ through a group homomorphism. This is called a representation. It must respect the group properties as every homomorphism does
$$
f_g(f_h(x)) = f_{g\cdot h}(x)\,.
$$
Representations in terms of functions are useful because you can actually do something with the group elements. Even more useful are those where the functions are just matrices. But you are not forced to do this, at least conceptually.
Second question
Invertibility is a very natural assumption. A symmetry is a transformation that leaves an object invariant.$^1$ The only sensible way to define the meaning of invariant is through an equivalence relation. Namely $A$ is invariant under $g\in G$ if
$$
g\cdot A \sim A\,,
$$
where $\sim$ is an equivalence relation. In particular, it is symmetric. I don't think any notion of invariance that doesn't use an equivalence relation can deserve such a name.
Due to the symmetry of this relation, it's natural to think of $g\cdot A$ as $A'$ and assume the existence of a transformation that maps $A'$ to $A$, since they are supposed to be equivalent. Therefore
$$
h \cdot A' = A = h \cdot g \cdot A\,,
$$
thus $h\cdot g$ acts as the identity transformation.

${}^1\;$ For example, the group $D_3$ is the set that leaves the equilateral triangle invariant.
A: They needn't.
People have definitely been interested in "non-invertible symmetries". We typically don't call them "symmetries", i.e., we reserve the name "symmetry" for those operations that are invertible. In this sense, it would be better to say that a symmetry is, by definition, an operation that is invertible; but non-invertible operations are interesting too.
In the quantum context, these are referred to as defects, borrowing terminology from the condensed-matter community. A symmetry is an invertible defect. But not all defects have to be invertible. And understanding the non-invertible defects a given system may support is just as powerful, if not more, than understanding its symmetries. It's just that, well, it is harder.
The case where these objects were studied first is, to the best of my knowledge, conformal field theories. Here the defects are elements of a fusion category (the relevant algebraic structure is better understood as a ring rather than a monoid, as in the OP, but their guess was not too far off). The abelian subcategory corresponds to invertible defects. But the whole category can be studied, no need to restrict the attention to the abelian objects only. A quick google search yields Duality Defect of the Monster CFT as a recent example where non-invertible symmetries are studied in quite some detail.
But anyway, this is just to show that non-invertible operations are extremely interesting, without a doubt. We don't usually find these in introductory textbooks because the mathematics are more challenging. But people do care.
A: Symmetries are a bit deeper than just functions or maps.  They actually act upon functions or maps.  This answer mentions a category theory definition for symmetries:

A symmetry of a morphism $\phi:A\to B$ means a pair $(\alpha,\beta)$ of automorphisms of $A$ and $B$ respectively, such that $\beta\circ\phi=\phi\circ\alpha$.

Of course, category theory is impenetrable if one doesn't already know it.  The general idea is that a symmetry is captured not just in the mapping of sets, but of the structure of those sets.  If I have a triangle with vertices A, B, and C, it not only maps A B and C to A' B' and C', but also maps the relationships between them.  If A and B are connected, A' and B' are connected.  And it doesn't matter if you apply the relationship first and then map the result with $\phi$, or if you start with mapping with $\phi$ and then apply the relationship.  A map has such a relationship with the terrain it represents.  It doesn't matter if you walk 10 miles north, and then find the point where you are, or if you start by seeing how many map-inches 10 miles is, and then walk the corresponding amount.  It doesn't matter if you draw a 5" line to the left, starting at your nose, or draw a 5" line to the right of your mirror image.  The result is the same.
As for why they have an inverse, that's more of a question of why did we name "symmetry" "symmetry."  They have it because we found a useful class of things to talk about had these behaviors.  We could talk about something which does not need to be an isomorphism (which would mean it does not need to have an inverse).  It just turned out that the things that had inverses were far more useful to talk about.  Indeed, there is an article out there, Semigroup Theory of Symmetry, which goes into detail about this more generalized concept of symmetry, and even goes beyond this to look at an even more concept: groupoids of symmetry.  (Unfortunately, this is behind a paywall, so I have not actually been able to access it.  You might!)
In a very vague sense, we are interested in mappings that preserve all structure.  To say that, we typically need to be able to map it back, to show that the structure did not change.  If we were to remove this requirement, we could have "symmetries" involving relationships that can never be demonstrated to preserve structure.   That just turned out not to be that useful.
But let's widen the net.  Let's consider what I will call semi-symmetries, which obey the above definition except using endomorphisms rather than automorphisms.  What sort of structures do we see.
In the finite case, where these endomorphisms are just functions, we see that the endomorphisms must change the cardinality of the set we are looking at.  If the cardinality of the domain and range are the same then the mapping must be bijective, and this is an invertable function (an automorphism).  So in the finite case, any semi-symmetry operation is going to involve either expanding or contracting the number of objects which can be explored.  In a sense, these maps may be "given groups of people such that everyone is married to one other, map people to the taller person in the married couple."  They can be done, but they lack the feel of "symmetry."
So what about the infinite cases?  I came across one very interesting example of this sort of semi-symmetry: Hilbert's Grand Hotel.  This is a very common paradox that one has to work through when learning set theory to understand the counter-intuitive math to resolve the paradox.  The problem looks like this:

Hilbert has a Grand Hotel with an infinite number of rooms, each numbered with a natural number.  A new client shows up, looking for the room, but is dejected because the sign out front reads "No Vacancy."  Hilbert assures the client that this is not a problem, and gives instructions to the bellhops to ask each party to please pack up, and move into the room with a room number one higher than they were originally in.
All the guests oblige, and now room 0 is empty.  Hilbert happily books the new client, and hands them the keys.

In this example, there is a semi-symmetry.  $\alpha$ is "every client moves to a room one higher in number" and $\beta$ is $n^\prime = n + 1$.  Its easy to see that the $\beta$ equation is non-invertible over natural numbers, because there is no $n$ such that $n^\prime = 0$.
This problem is interesting because it highlights that there are some injective mappings in set theory which do not change the cardinality of the set.  I would argue that this Hilbert Hotel situation would earn the intuitive sense of a "symmetry," although the actual math shows it to be a semi-symmetry.  This would suggest that there is some validity to thinking of at least some of the non-invertible mappings as being part of a symmetry.
So having shown that it makes some sense, we are left with the question of "why not?"  There must be a "soft" reason for why we preferred to focus on the invertible mappings and not the non-invertible ones.  The best answer I found for this is from another stack exchange question focusing on the underlying algebraic structures: Why are groups more important than semigroups?.  In the wining answer:

One thing to bear in mind is what you hope to achieve by considering the group/semigroup of automorphisms/endomorphisms.
A typical advantage of groups is that they admit a surprisingly rigid theory (e.g. semisimple Lie groups can be completely classified; finite simple groups can be completely classified), and so if you discover a group lurking in your particular mathematical context, it might be an already well-known object, or at least there might be a lot of known theory that you can apply to it to obtain greater insight into your particular situation.
Semigroups are much less rigid, and there is often correspondingly less that can be leveraged out of discovering a semigroup lurking in your particular context. But this is not always true; rings are certainly well-studied, and the appearance of a given ring in some context can often be leveraged to much advantage.
A dynamical system involving just one process can be thought of as an action of the semigroup N
. Here there is not that much to be obtained from the general theory of semigroups, but this is a frequently studied context. (Just to give a perhaps non-standard example, the Frobenius endomorphism of a char. p ring is such a dynamical system.) But, in such contexts, precisely because general semigroup theory doesn't help much, the tools used will be different.

To sum this quote up: groups provide a very powerful platform for deriving insight from these relationships.  Semigroups are simply not as powerful, making it harder to make useful inferences.  Of course, given that semigroups naturally occur in dynamic systems, there are clearly useful semi-symmetries to be had.  However, they are not useful because they are semi-symmetries, but rather useful because of their association with dynamic systems.  Accordingly, their value is not phrased using the language of symmetries, but rather in the language of dynamic systems.  The fact that they are semi-symmetries is more of an afterthought.  Similarly, from what I can glean from the research, semi-symmetries are interesting for those who study rings, but those analyses are done with respect to the additional structure a ring provides.
