In Classical Mechanics we usually describe the possible configurations of a system by points on a smooth manifold $M$ which is the configuration manifold of the system.

In that case, when we talk about transformations of the possible configurations of the system we usually do so introducing diffeomorphisms and one-parameter families of diffeomorphisms. The infinitesimal version of those transformations are then described by vector fields on the configuration manifold. The generator of a one-parameter family of transformations is then a vector field.

Now, in Quantum Mechanics we describe the possible states of the system by vectors in a Hilbert space $\mathcal{H}$. I'm unsure, however, how do we describe those "transformations" (for example, translations and rotations) like we do in Classical Mechanics.

Since we always want to preserve the linear structure it is clear those transformations should be linear operators in $\mathcal{H}$. But it is only this that we require?

Also, how does one describe the "generator of a certain transformation" as we do in Classical Mechanics? We don't have vector fields here to do this, so I'm unsure how to describe it.

In that case, from a mathematically rigorous standpoint, how does one, in analogy with Classical Mechanics, describe transformations of states corresponding to actual transformations of the systems? And how do we describe the infinitesimal version of those transformations properly?