Are there any physical symmetries that do not form a group? I'm trying to understand the role of symmetry in physics. The overall derivation seems pretty intuitive, reasonable and elegant, but one point I'm curious about is the identification of physical symmetries with groups. It does not seem a priori necessary that the transformations under which a mathematical object remains invariant obey the group axioms of associativity and invertibility. Identity seems necessary, since leaving the object unchanged obviously should not produce a change in the invariant, but it is at least conceivable that invariants can exist under non-associative or non-invertible transformations. Are there any examples of such invariants?
 A: Symmetries act on objects. For a symmetry to be nonassociative, it would have to be the case that (doing A and then doing B) and then doing C was different from doing A and then (doing B and then doing C). I don't think that's possible.
There is a previous question about invertibility with some good answers. In short, there are noninvertible analogs of symmetries, but they often aren't called symmetries (i.e., "symmetry" is taken to imply invertibility by definition).
A: Consider a transformation that cuts out a unit square centred on the origin from the plane, throws away the rest of the plane, and magnifies the unit square by a factor of two. This is non-invertible, since the step of "throwing away" everything outside the unit square cannot be reversed. But there are lots of subsets of the plane that are invariant under this operation.
Perhaps the most obvious shape with this symmetry is a 2x2 square centred on the origin. We cut out square piece from the middle of it and magnify it to exactly replace the whole. But there are plenty more.
Define an infinite sequence of nested squares, starting with the 2x2 square, then 1x1, (1/2)x(1/2), etc. each one half the size of the previous. Draw any pattern in the region between the first and second squares, then copy it scaled down into each subsequent region. The result is a fractal that is invariant under the described transformation.
There are many fractal shapes in nature that look like this. Clouds, fern leaves, snowflakes, trees, lungs, river drainage basins, etc. We commonly say that they have a scale invariance - if you magnify them they look the same. But of course they don't, because if they did they would have to be of infinite size, as you scaled them up. What we mean is that pieces of them scaled up look the same as the whole thing. 'Taking a piece' is a non-invertible transformation.
It's usually mathematically easier to consider the corresponding invertible transformation and only cut out our finite piece from the infinite one once we're done. Groups are easier to work with. But whenever we talk about "pieces" of a shape, we're using non-invertible transformations.
Similarly, solid state physicists normally assume a crystal lattice extends infinitely - or use cyclic boundary conditions to get the same effect, so they can use groups. As any real crystal is finite, we're always really talking about pieces being similar. A finite crystal is still considered to have some sort of translational symmetry, even though this is impossible.
For another example, consider the famous Mandelbrot set and the related Julia sets, which are invariant under a transformation translating and squaring numbers in the complex plane. Squaring is non-invertible - there are two points mapping to any given point.
A: I wanted to add a simple, physical example, of non-grouplike symmetries to the existing answers, in order to answer to the exact title of your question.
Consider the critical Ising model in two dimensions. It has three primary local operators, $\mathbf{1}$, $\sigma(x)$ and $\epsilon(x)$, representing the identity operator, the spin, and the energy, respectively. Their OPE is
$$\begin{aligned} 
\sigma\times\sigma&\sim \mathbf{1}+\epsilon \\
\sigma\times\epsilon&\sim \sigma \\
\epsilon\times\epsilon&\sim \mathbf{1}.
\end{aligned}$$
Now the Ising model has a celebrated $\mathbb{Z}_2$ symmetry. As explained nicely in @ConnorBehan's answer, associated to this symmetry there is a topological operator $U$ (I suppress the dependence of $U$ on the codimension-1 manifold (here, curve)). $U$ acts on our local fields, by wrapping around them, as follows:
$$\begin{aligned}
U\,\sigma &= -\sigma \\
U\,\epsilon &= \epsilon \\
U\,\mathbf{1} &=\mathbf{1},
\end{aligned}$$
i.e. it flips the spin, and leaves the rest untouched. Since $U$ is the generator of a $\mathbb{Z}_2$ symmetry, it squares to one. In topological operator language, it fuses with itself to the identity, $U^2\equiv U\otimes U = \mathcal{I}$, where $\mathcal{I}$ is the identity topological line operator (a transparent line). (Note: $\mathcal{I}\neq \mathbf{1}$, one is a line operator, the other one a local operator)
Notably, the Ising model has another symmetry, oftentimes called a duality, namely, the Kramers-Wannier symmetry. At the level of topological operators, this is expressed by means of an operator $K$, acting on the fields as
$$\begin{aligned}
K\, \sigma &= 0 \\
K\, \epsilon &= -\color{gray}{\sqrt{2}}\,\epsilon \\
K\, \mathbf{1} &= \color{gray}{\sqrt{2}}\,\mathbf{1}.
\end{aligned}$$
Now you can ask what happens if I do $K$ twice? Or $K$ after $U$? Or $U$ after $K$. What you will find is that the only fusion rules compatible with the OPE above are the following:
$$\begin{aligned}
U\otimes U &=\mathcal{I} \\
K\otimes U = U\otimes K &= K \\
K\otimes K &= \mathcal{I} + U.
\end{aligned}$$
I.e. the symmetries of the Ising model do not form a group!
Mathematically, this structure is known as a $\mathbb{Z}_2$ Tambara-Yamagami category :))
A: Mathematically speaking, you are right, there is no necessity for the invertibility axiom or the associativity axiom, or both. If you drop the first you get a monoid, if you drop both, you get a non-associative monoid. You could also drop the identity which gives you a magma and if you then reinstate associativity, you get a semigroup. You can dualise and also get comonoids and comagmas etc.
There is in fact, a whole zoo of mathematical structures in physics that mathematical physicists are busy discovering, including the above. For example, the manifolds that general relativists are so fond of can dually be thought of certain algebras. Here, 'algebras' does not mean the generic term but is a specific term referring to a specific structure, essentially a vector space where multiplication is possible.
Symmetries in physics are identified with groups essentially because of Noether's theorem which shows that symmetries yield conservation laws and vice-versa. I'm not aware of a Noether theorem where the group axioms are weakened – but there could well be – you could try asking that in MathOverflow.
Mathematical structures, in a philosophical sense, represents symmetries in a larger sense. For example, algebras represent manifolds and in manifolds, all points are alike, and this is a kind of symmetry usually called homogeneity.
A: There is something called a non-invertible symmetry which sounds like it's exactly what you're looking for. A place to read about it is a recent paper on 2d QCD.
To summarize the motivation, an ordinary symmetry acts on a local operator as
\begin{equation}
\mathcal{O}(x) \mapsto U(g) \mathcal{O}(x) U(g)^{-1}
\end{equation}
where $U(g) \times U(h) = U(gh)$. If the symmetry is continuous, $U(g)$ is the exponential of a charge. But to get a charge, we integrate the zeroth component of a conserved current. Therefore,
\begin{align}
\mathcal{O}(x) &\mapsto e^{Q} \mathcal{O}(x) e^{-Q} \\
&= \mathcal{O}(x) + [Q, \mathcal{O}(x)] + \dots \\
&= \mathcal{O}(x) + \int_{t - \epsilon} J^0(y) \mathcal{O}(x) d\textbf{y} - \int_{t + \epsilon} \mathcal{O}(x) J^0(y) d\textbf{y} + \dots
\end{align}
now follows. The key step is that, because the current is conserved, if we just take $J^0$ and integrate it over a closed surface, we will get zero. The difference of integrals above (one over space at time $t + \epsilon$, the other over space at time $t - \epsilon$) form a closed surface except one which contains the point $x$ for the operator. Therefore this surface can be deformed into any desired shape (i.e. it is like a Gaussian surface) and we say that the action of the symmetry is topological.
Generalizations of symmetries now come from running with the above abstract principle and forgetting about Noether currents. A non-invertible (0-form) symmetry of a theory in $d$ dimensions is a nonlocal operator $L$ whose action on local operators is topological. That is, if we let $L$ live on a $d-1$ dimensional surface and $\mathcal{O}$ live on a point, $L[\Sigma] \mathcal{O}(x)$ is independent of everything except the side of $\Sigma$ where $x$ can be found. These things are sometimes as useful as symmetries because they can be gauged and are preserved by interactions. They form a fusion category meaning
\begin{equation}
L_i \times L_j = \sum_k N^k_{ij} L_k
\end{equation}
which is more general than the group law.
A: This started out as a comment on benrg's answer (which I like and fully agree with), but I felt it deserved to be an answer in itself.
I would argue that associativity is a (desirable) mathematical artifact that we impose on transformations because we want them to be composable.
And, being a mathematical artifact, it cannot have a physical meaning.  Essentially, symmetries in nature aren't associative — our mathematical models of them are.

Basically, let's say that we have a physical system whose possible states we can somehow represent as mathematical objects of some kind.  We can then represent transformations between different physical states by functions that map these objects to each other.  And since the result of applying a transformation to a physical state is itself a physical state, we can then apply another transformation to it, and then yet another, and so on.
However, what we don't have, a priori, is any reason to believe that there should be any way to meaningfully combine two or more consecutive transformations into one transformation that has the same effect.  That is to say, if $x$ is a physical state and $f$ and $g$ are transformations, then we can say (by construction of our model) that $y = f(x)$ is also a physical state, and so is $z = g(y) = g(f(x))$.  But we don't have any real reason to believe that there should exist any physically meaningful transformation $t$ such that $z = t(x)$!
However, mathematically, we can simply define one to exist anyway!  Function composition, after all, is mathematically well defined for any two functions acting on the same space of objects, so we can simply decide that, for any two transformations $f$ and $g$, there also exists in our model a transformation $t = g \circ f$ representing the act of first applying $f$ and then $g$.
Does $t$ have physical meaning?  Well, insofar as applying two transformations one after another, and calling the whole process a single transformation, makes sense, then it does.  But in general, we have no reason to expect it to have any other meaning, or even for $t$ to have any mathematical representation that would be more natural than "$f$, then $g$".
Nonetheless, we automatically get associativity, because function composition is associative.  And even if we were working in a mathematical framework that didn't have a suitable notion of function composition defined already, we'd still want to define our notion of "transformation composition" to be associative.
Why?  Simply because, if $t = g \circ f$ and $u = h \circ g$ are both composite transformations, then we want both $h \circ t = h \circ (g \circ f)$ and $u \circ f = (h \circ g) \circ f$ to represent the same transformation mapping $x$ to $w = h(g(f(x)))$, i.e. the act of first applying $f$, then $g$, and then $h$.
And because this is how we want the composition of transformations to behave, then even if we were defining transformation composition axiomatically from the ground up (and thus had full freedom to choose how it should work), the associative law $h \circ (g \circ f) = (h \circ g) \circ f$ should be one of the axioms that we'd want to include, just because it makes composite transformations combine together in the manner that representations of "two or more transformations applied in sequence" should combine.
