Is it correct to say that in QM Operators are a way of representing elements of a group acting on a state, as linear maps on a Hilbert space? Sorry if my question is nonsense, but I was reading about representation theory in Quantum mechanics and I find it very interesting. As I understood, an example would be the charge operator, which is the generator of the $U(1)$ action by unitary transformations on the state space $H$. But something that I still get confused is how the elements of a group can act on a quantum mechanical state the same way as a operator would. I think my basics concepts are probably quite wrong or I am missing something here.
 A: This is actually a very important idea which, in my opinion, tends to be under-emphasized.
First, some background. Given a group $G$ and any set $X$, a (left) group action $\triangleright:G\times X \rightarrow X$ is a map such that:

*

*The action of the identity element $e\in G$ on any element $x\in X$ is given by $e\triangleright x = x$.

*For any $g,h\in G$, we have $g\triangleright\big(h\triangleright x\big) = (gh)\triangleright x$, so acting with $g$ after acting with $h$ is equivalent to acting with $gh$.

A right action $\triangleleft:X\times G \rightarrow X$ is defined analogously, but which obeys the slightly different composition property that $\big(x\triangleleft h\big)\triangleleft g = x\triangleleft(hg)$, so acting with $g$ after acting with $h$ is equivalent to acting with $hg$.
Both left-actions and right-actions are useful constructions which provide us with a mechanism for acting on arbitrary sets with the elements of a group.  The relevant set to this question is the set of (pure) physical states of a quantum mechanical system, which can be understood as the projective Hilbert space $\mathcal P(\mathscr H)$ associated to the Hilbert space $\mathscr H$ underlying the theory.  Specifically, $\mathcal P(\mathscr H)$ is the set of equivalence classes of vectors in $\mathscr H$ under the equivalence relation $\psi \sim \phi \iff \psi = \lambda \phi$ for some non-zero complex number $\lambda$.
Crucially, $\mathcal P(\mathscr H)$ is not a vector space but does have infinite cardinality (except in the trivial cases of $\mathrm{dim}(\mathscr H)<2$).  This presents us with a problem, namely that in order to specify the left-action of some group $G$ on $\mathcal P(\mathscr H)$, we would need to find a way to specify it on an uncountably infinite set, and this is generally very difficult.
A very good solution is to not consider the action of $G$ on $\mathcal P(\mathscr H)$ directly. Instead, we consider the action of $G$ on $\mathscr H$ first, which then induces an action of $\mathcal P(\mathscr H)$.  More specifically, we consider an action of the form
$$g \triangleright \psi = \rho(g) \psi$$
where $\rho$ has the following properties:

*

*$\rho(g)\rho(h) = c_{gh}\rho(gh)$ for some $c_{gh}\in \mathbb C$

*$\rho(g)$ is either a linear or antilinear map on $G$
These conditions make $\rho$ a projective linear (or antilinear) representation of $G$ on $\mathscr H$. It's not hard to verify that if this is the case, then the corresponding action induces a well-defined action on $\mathcal P(\mathscr H)$ insofar as it is compatible with the equivalence relation $\sim$, and therefore maps equivalence classes to equivalence classes. Furthermore, it is quite economical; we need only define the action $\triangleright$ on a basis of $\mathscr H$, and then let it extend to general elements of $\mathscr H$ by (anti)linearity.
The all-important Wigner theorem tells us that every symmetry transformation on $\mathcal P(\mathscr H)$ can be induced by either a unitary or antiunitary operator on $\mathcal H$. As such, when we want to define the action of a symmetry group $G$ (e.g. the rotation group $\mathrm{SO}(3)$) on $\mathcal P(\mathscr H)$, what we should do is try to find projective unitary or antiunitary representations of $G$ on $\mathscr H$.


But something that I still get confused is how the elements of a group can act on a quantum mechanical state the same way as a operator would.

As I mentioned above, we define the action of a group element on a vector $\psi\in \mathscr H$ through a (projective) representation $\rho$ via
$$g \triangleright \psi = \rho(g) \psi$$
That is, we associate to the group $G$ a set of operators $\rho(G)$ which implement its action on vectors in $\mathscr H$. In this way, we allow $G$ to act on $\mathscr H$, and by extension on $\mathcal P(\mathscr H)$.
If $G$ is a symmetry group, then the family of operators $\rho(G)$ are either unitary or antiunitary.  Of particular importance is the unitary representations of Lie groups; in this case, $\rho(G)$ constitutes a set of unitary operators which implement a symmetry, and the induced representation $\mathrm d\rho$ of the Lie algebra $\mathfrak g$ yields a set $\mathrm d\rho(\mathfrak g)$ of self-adjoint operators which represent the conserved charges associated to that symmetry c.f. Stone's theorem.
As a concrete example of the above, rotations are implemented as symmetry transformations on $\mathcal P(\mathscr H)$ via projective unitary representations $\rho$ of $\mathrm{SO}(3)$ on $\mathscr H$.  The operators $\rho\big(\mathrm{SO}(3)\big)$ are the rotation operators on $\mathscr H$, and the self-adjoint operators $\mathrm d\rho\big(\mathfrak{so}(3)\big)$ are the angular momentum operators.
In the same way, spatial translations are implemented via representations of $\mathbb R^3$ on $\mathscr H$. The operators $\rho(\mathbb R^3)$ are the translation operators, and the self-adjoint operators $\mathrm d\rho\big(\mathbb R^3\big)$ are the momentum operators. Furthermore, time translations are implemented by representations of $\mathbb R$, where $\rho(\mathbb R)$ are the propagators and $\mathrm d\rho(\mathbb R)$ yields (up to some multiplicative constant) the Hamiltonian.
