From Shankar's QM book pg. 310, it was said that the quantum hamiltonian $H$ is rotationally invariant whenever the classical hamiltonian $\mathcal{H}$ is rotationally invariant.
For infinitesimal rotations, the classical variables $(x,y,p_x,p_y)$ transform as $$\bar{x}=x-y\epsilon$$ $$\bar{y}=y+x\epsilon$$ $$\bar{p}_x=p_x - p_y\epsilon$$ $$\bar{p_y}=p_y+p_x\epsilon$$ The classical hamiltonian $\mathcal{H}$ is rotationally invariant if $H(\bar{x},\bar{y},\bar{p}_x,\bar{p}_y)=H(x,y,p_x,p_z).$
The quantum operators $(X,Y,P_x,P_y)$ transform in a similar way: $$U^\dagger XU=X-Y\epsilon$$ $$U^\dagger Y U = X\epsilon + Y$$ $$U^\dagger P_x U=P_x - P_y\epsilon$$ $$ U^\dagger P_y U=P_x\epsilon+ P_y $$
where $U$ is the infnitesimal rotation operator. The quantum hamiltonian is rotationally invariant if $H(U^\dagger X U,U^\dagger Y U,U^\dagger P_x U,U^\dagger P_y U)=H(X,Y,P_x,P_y)$.
Now since the operators $X,P_x$ and $Y,P_y$ do not commute like classical variables $x,p_x$ and $y,p_y$, how can we be sure that the quantum hamiltonian is invariant whenever the classical hamiltonian is?
For example consider that in the expansion of classical Hamiltonian $\mathcal{H}(\bar{x},\bar{y},\bar{p}_x , \bar{p}_y)$ we have two terms that cancel each other: $$\mathcal{H}(\bar{x},\bar{y},\bar{p}_x , \bar{p}_y)=...+xp_x-xp_x+...=\mathcal{H}(x,y,p_x,p_y)$$
How can we be sure that in the expansion of the quantum Hamiltonian $H(U^\dagger X U,U^\dagger Y U,U^\dagger P_x U,U^\dagger P_y U)$ these two terms will be $$H(U^\dagger X U,U^\dagger Y U,U^\dagger P_x U,U^\dagger P_y U)=...+XP_x-X P_x+...=H(X,Y,P_x,P_y)$$ which cancel instead of $$H(U^\dagger X U,U^\dagger Y U,U^\dagger P_x U,U^\dagger P_y U)= ... + XP_x - P_xX+...\neq H(X,Y,P_x,P_y)$$ which do not cancel?