Quantum Mechanics of a rigid body that has only finitely many possible axes of rotation I consider the quantum version of dynamics of rigid body motion with intrinsic angular momentum, having moment of inertia $I$. Given a classical free system of one free rotating spherical rigid body (i.e. $I = I_0 \mathbb{1}_3$) in the Hamiltonian formulation:
\begin{equation}
H = \frac{\vec{L}\cdot \vec{L}}{2I_0}
\end{equation}
where $I$ is the moment of inertia, the quantised description by canonical quantization is obtained by simply promoting the coordinates in the classical description ($\vec{\theta}, \vec{L}$) to operators with properties from the Poisson brakets translated:
\begin{equation}
\{L_i, L_j\} = \varepsilon_{ijk}L_k \mapsto [\hat{L}_i, \hat{L}_j] = i\hbar\varepsilon_{ijk}\hat{L}_k \qquad etc.
\end{equation}
Thus the operators $\hat{L}_k$ obey the commutation relation of the Lie algebra $\mathfrak{su(2)}$. With the postulate that $\hat{L}_k$ needs to be an observable, $\hat{L}_k$ is Hermitian and there must be a basis of eigenvectors of $\hat{L}_k$. One can then construct these as eigenvectors of $\hat{L}_z$, with labels $|j, m\rangle$. This is just standard quantum mechanics. But now:
1.Question: I had not specified the accessible phase space of the classical spinning top, because I assumed the top can spin in any direction of $\mathbb{R}^3$ and with any magnitude $||\vec{L}||$. How would the quantum description change if I restrict the classical phase space, to lets say only discrete directions like $\pm \vec{e}_x,\pm \vec{e}_y,\pm \vec{e}_z$ (so it only spins in 6 directions) or more generally, any discrete subgroup of $O(3)$?
2.Question: If one identifies the the direction of the rotation axis $\vec{n} \parallel \vec{L}$ with a matrix $U \in O(3)$ that rotates space around $\vec{n}$ by magnitatude $||\vec{L}||$, one could write just as well $H = c \cdot \text{Tr} \ U^\dagger U $, with the Frobenius inner product. A generalisation would be to let $U\in U(3)$ etc. If one quantizes now, would the operator $\hat{U}$ still be Hermitian?
 A: It seems to me that there are some difficulties with your framing of the problem. The language in the question about tops suggests that you have in mind the quantum treatment of a rigid rotor, whose moment of inertia is fixed in the body-fixed frame (and is diagonal in the principal axis coordinates). In your classical Hamiltonian, you take the moment of inertia to be a scalar, but this only works when the body has spherical symmetry. In general, the moment of inertia is a tensor, and then the Hamiltonian can't be written in the form you give.
So your question seems to be asking about the quantum mechanics of the rotation of a sphere. But quantum-mechanically, a sphere has no collective rotations. This is why we don't observe rotational bands in spherical nuclei, such as nuclei with magic numbers of neutrons or protons.
If you want to fix this problem by breaking the spherical symmetry, and you make your object a rigid rotor, so that its moment of inertia has a fixed form in the body-fixed frame, then you get the well-studied properties of rotational bands in nuclei and molecules. For example, a parity-symmetric rotor like the H2 molecule will have a spectrum of rotational states with even parity, even spin, and J(J+1) spacing of the energy levels. In nuclei, the keyword you want is "particle-rotor model," and there you get some non-obvious results such as the different behavior of bands in odd nuclei with ground-state spin 1/2.
