Do group elements correspond to observables of a system, or to transformations of a system?

Question

Most groups I encountered so far in classical mechanics an special relativity (Galilei-group, Poincare-group, Lorentz-group and so on) described transformations of the system that was to look at. This is somehow intuitive to me, because a transformation succeeded by another transformation is of course a transformation again.

However (and this is something I can't really wrap my head around): In QM, when talking about representations, it seems to me that choosing a representation maps group elements to operators, that is, to observables of the system. Which is of course less intuitive to me (given two positions, why would they have any "group" structure? Of course, if I imagine them to be vectors, I could just add them, but is this what is meant by position becoming a group element now?)

Which is true now? Group elements corresponding to transformations, or to observables? Is there an intuitive approach in how to to group elements are related to observables in EVERY theory that is based on groups?

Edit: I know that in QM, Observables are seen as Operators of an Algebra that act on the Hilbert Space. And of course, those Operators do form a group as well (because linear operators do form the general linear group). But this group is not always the same group that "the theory is built uppon"

Answer 1

In quantum mechanics, not every operator is an observable. Observables correspond to Hermitian operators. Not all operators are Hermitian; a rather well-known example of non-Hermitian operators are the ladder operators familiar from the harmonic oscillator or quantum field theory. Since these are not Hermitian, they are not observables.

Actions of groups on quantum states are usually given by unitary operators. In general, these are not Hermitian. However, things are interesting if our group is a Lie group, a group depending smoothly on some parameters. Such groups include rotations (which depend smoothly on the rotation angles) and translations. From the theory of Lie groups, it is known that a Lie group action can be written as the exponential of a generator, which can be seen as an 'infinitesimal group transformation'.

So suppose that we have an action of the rotation group SO(3) on our quantum state given by $|\psi\rangle \rightarrow U |\psi \rangle$, where $U$ is the operator doing a rotation about the z-axis (for definiteness), then we know that we can write $U = e^{i \theta J_z}$ where $J_z$ is the generator of rotations around the z-axis. For small $\theta$, we can also write $U \approx 1 + i \theta J_z$ if we like. Now, it turns out that $J_z$ is Hermitian and thus an observable; it corresponds to the z-component of angular momentum, although my normalization is probably incorrect. The observable can be seen as quantifying how much a state changes upon applying a rotation. In general, group generators (but not transformations) correspond to observables.

There are exceptions to the rule that unitary operators are not Hermitian. A relevant example is the (unitary and Hermitian) parity operator. This can be seen as an action of the group $\mathbb{Z}_2$ on quantum states. It has eigenvalues $\pm1$ and is an observable. Note that $\mathbb{Z}_2$ is a discrete group and therefore not a Lie group in any interesting way, so the above discussion does not hold here.