Is $U^\dagger(R)\hat{H}U(R)=\hat{H}$ always true? Consider a Rotation transformation on momentum state,
$$U^\dagger(R)\hat{\mathbf{p}}U(R)=R\hat{\mathbf{p}}$$
Now the question is whether,
$$U^\dagger(R)\hat{H}U(R)=\hat{H}\,?$$
Here, $\hat{H}$ is the Hamiltonian of a free particle. Is it always true? Is there any counter examples?
My attempt:
\begin{align}
U^\dagger(R)\hat{H}U(R)&=\frac{1}{2m}U^\dagger(R)\hat{\mathbf{p}}^2U(R)\\
&=\frac{1}{2m}U^\dagger(R)\hat{\mathbf{p}}U(R)U^\dagger(R)\hat{\mathbf{p}}U(R)\\
&=\frac{1}{2m}(R\hat{\mathbf{p}})(R\hat{\mathbf{p}})
\end{align}
Is this always true that $$\frac{1}{2m}(R\hat{\mathbf{p}})(R\hat{\mathbf{p}})=\frac{1}{2m}\hat{\mathbf{p}}^2\, ?$$
If it is why? If not when it is not?
Note: This is an exercise from Coleman's course 253a (https://arxiv.org/abs/1110.5013). See equation (1.8) there. It would be better if the answer is provided using his notations.
 A: It is true in this specific case that, if $\hat H=\frac{1}{2m}\mathbf{p}^2$ is the Hamiltonian for the free particle, then $U^\dagger\hat{\mathbf{p}}^2 U=\hat{\mathbf{p}}^2$ for $U$ a rotation.
Maybe the 2d case is sufficient to illustrate the point.  We have
\begin{align}
R^\dagger_z(\theta)\hat{p}_xR(\theta)&=\hat{p}_x^\prime=\hat{p}_x\cos\theta+\hat{p}_y\sin\theta\, ,\\
R^\dagger_z(\theta)\hat{p}_yR(\theta)&= \hat{p}_y^\prime=-\hat{p}_x\sin\theta +\hat{p}_y\cos\theta
\end{align}
and then
\begin{align}
\mathbf{p^\prime}^2=\mathbf{p}^2\, .
\end{align}
Note that
\begin{align}
{\hat{p}_x^\prime}^2=\hat{p}_x^2+\hat{p}_y^2 +\hat{p}_x\hat{p}_y
+\hat{p}_y\hat{p}_x
\end{align}
keeping the ordering straight but since $\hat{p}_x$ and $\hat{p}_y$ commute then it boils down to the usual $\hat{p}_x\hat{p}_y+\hat{p}_y\hat{p}_x=2\hat{p}_x\hat{p}_y$.
A: If the operator $U$ is indeed the unitary operator, then
$U U^{\dagger}= U^{\dagger} U = 1$
then expression you have (which appears to be applied incorrectly) above will reduce back to its original form
$\frac{1}{2m} U^{\dagger} p^2 U = U^{\dagger} H U = H$
A: Base transformations of a scalar operator (H) and the vector operator's spatial rotation are confused here, I think. Try to rewrite it in bra-ket notation.
Coordinate -> momentum space transformations also done in $ UHU^{-1} $ way and it really changes H.
A: There is a dot product in fact $\mathbf p^2 = \mathbf p^{\dagger} \mathbf p$. So one may do the following:
$$
U^{\dagger} \mathbf p^{\dagger} \mathbf p \  U = 
U^{\dagger} \mathbf p^{\dagger} U U^{\dagger} \mathbf p \  U = (U^{\dagger} \mathbf p \  U)^{\dagger} U^{\dagger} \mathbf p \  U = (R \mathbf p)^{\dagger} R \mathbf p = \mathbf p R^{\dagger} R \mathbf p = \mathbf p^{\dagger} \mathbf p
$$
Here I've denoted the transposed momentum $p^{\dagger}$ to unify notation, despite the fact, it is real, And in the last step, I've used, that the $R$-matrix is unitary.
