How the quantum Hamiltonian changes under a transformation? Let's say that I have an Hamiltonian $H(k)$ in momentum space and I consider a transformation (to be concrete let's say time reversal  $\mathcal{T}$). We say that this is a symmetry, if
$$\mathcal{T} H(k) \mathcal{T^{-1}}=H(-k).$$
My question is (in general): Why does the Hamiltonian transforms under a generic transformation $\mathcal{T}$ as $\mathcal{T} H(k) \mathcal{T^{-1}}$? Where does this transformation law come from?
What confuses me about this is that I thought that under a change of basis $\psi \rightarrow U\psi$ the hamiltonian (or any other operator) would change according to $U^{\dagger}HU$. So what is going on?
 A: One can always define a second unitary operator $V=U^\dagger$ to write
$$U^\dagger H U= V H V^\dagger.$$ This actually holds true for all of group theory, where by definition the inverse of a group element must also be a member of the group. As such, we can equally well inspect the set of transformations
$$\mathfrak{g}^{-1}\mathfrak{h}\mathfrak{g}$$ or $$\mathfrak{g}\mathfrak{h}\mathfrak{g}^{-1}$$ for a set of group elements $\mathfrak{g}$ because they will lead to the same overall set of transformations.
The important thing mentioned by OP is to match up the transformations on operators with transformations on things like quantum states (like rotating vectors versus rotating coordinate systems).  Indeed when a state undergoes the transformation $|\psi\rangle\to U|\psi\rangle$, this is equivalent to observables (operators) undergoing the transformation $O\to U^\dagger O U$. However, this is also equivalent to the density operator transformaing according to $\rho\to U\rho U^\dagger$, to ensure consistency for pure states $\rho=|\psi\rangle\langle \psi|$. So one has to keep track of what is being transformed in order to choose the correct group element $U$ or $V$.
Time reversal has the nice property that if you apply it twice you get back what you started with. I might assume that $$\mathcal{T}^2 H\mathcal{T}^{-2}=H$$ implies that $\mathcal{T}^2$ must be a phase times the identity matrix. It turns out that $\mathcal{T}^2=\pm 1$, so $$\mathcal{T}=\pm\mathcal{T}^{-1} \quad\Rightarrow\quad \mathcal{T} H\mathcal{T}^{-1}=\mathcal{T}^{-1} H\mathcal{T}.$$ This is similar to the parity operator in that it doesn't matter whether you apply the operator or the inverse of the operator, you get the same final result. [Intuitively, the inverse of time reversal is just time reversal (up to a phase, that part isn't intuitive), so it doesn't matter if you reverse the time or undo the time reversal.]
A: Under the action of a unitary operator $U$, all operators $O$ transform as $O \to U O U^{-1}$ and states transform as $|\psi \rangle \to U | \psi \rangle$. The transformation law for $O$ takes that form so that the state $O | \psi \rangle$ transforms in exactly the same way as $ | \psi \rangle$.
A: I would like to add the following description based on Wigner's theorem, as described on page 62 in the book "quantum mechanics a modern development":
Theorem (Wigner). Any mapping of the vector space onto itself that preserves the value of $|\langle\phi \mid \psi\rangle|$ may be implemented by an operator $U$:
$ \begin{aligned} |\psi\rangle \rightarrow\left|\psi^{\prime}\right\rangle &=U|\psi\rangle \\ |\phi\rangle \rightarrow\left|\phi^{\prime}\right\rangle=U|\phi\rangle \end{aligned} $  (3.1)
with $U$ being either unitary (linear) or antiunitary (antilinear).
The transformation of state vectors, of the form (3.1), is accompanied by a transformation $A \rightarrow A^ \prime$ of the operators for observables. It must be such that the transformed observables bear the same relationship to the transformed states as did the original observables to the original states. In particular, if $A\left|\phi_{n}\right\rangle=a_{n}\left|\phi_{n}\right\rangle$, then $A^{\prime}\left|\phi_{n}^{\prime}\right\rangle=a_{n}\left|\phi_{n}^{\prime}\right\rangle$. Substitution of $\left|\phi_{n}^{\prime}\right\rangle=U\left|\phi_{n}\right\rangle$, using (3.1), yields $A^{\prime} U\left|\phi_{n}\right\rangle=a_{n} U\left|\phi_{n}\right\rangle$, and hence $U^{-1} A^{\prime} U\left|\phi_{n}\right\rangle=a_{n}\left|\phi_{n}\right\rangle$. Subtracting this from the original eigenvalue equation yields $\left(A-U^{-1} A^{\prime} U\right)\left|\phi_{n}\right\rangle=0$. Since this equation holds for each member of the complete set $\left\{\left|\phi_{n}\right\rangle\right\}$, it holds for an arbitrary vector, and therefore $\left(A-U^{-1} A^{\prime} U\right)=0$. Thus the desired transformation of operators that accompanies (3.1) is
$A \rightarrow A^{\prime}=U A U^{-1}$. (3.2)
A: Unitary transformations are motivated by the fact that a pure phase of a quantum state is not observable since $|\psi|^2 = |\psi e^{i\varphi}|^2$. This means that the 'physics" doesn't change if we apply a unitary transformation to the system. Here physics refers to three related things

*

*the probability density should be unaffected $\langle \psi |\psi\rangle = \langle \tilde\psi |\tilde\psi\rangle$

*the eigenvalues of some operator $O$ should be unaffected: if $O|n\rangle = O_n |n\rangle$ then $\tilde{O}|\tilde{n}\rangle = O_n |\tilde{n}\rangle$

*any expectation value should be invariant $\langle \psi | O |\psi\rangle = \langle \tilde\psi| \tilde{O} |\tilde\psi\rangle$
Anything else isn't directly observable in QM. We start by transforming a state $|\psi\rangle$ with the prescription
$$|\psi\rangle \to |\tilde\psi\rangle = U|\psi\rangle .$$
Since $U$ is unitary $U U^\dagger = 1$ which immediately yields
$$
\langle \psi|\psi\rangle \to \langle \psi | U^\dagger U|\psi\rangle = \langle \psi |\psi\rangle
$$
Next we demand that the expectation value remains invariant
$$
\langle \psi | O |\psi\rangle \to \langle \psi | U^\dagger \tilde{O} U |\psi\rangle = \langle \psi | O |\psi\rangle
$$
For this to be true in general when the operators don't commute we require that $O$ transforms as $O \to \tilde{O} = U O U^\dagger$. With this transformation law for operators it is also obvious that the spectrum is unchanged because
$$
\tilde{O}|\tilde{n}\rangle = U O U^\dagger U |n\rangle = U O |n\rangle = U O_n |n\rangle = O_n U |n\rangle = O_n |\tilde{n}\rangle 
$$
As mentioned in the other posts the unitary operators form a group, if $U$ is unitary so is $U^{-1} = U^\dagger$ so it really is somewhat arbitrary how we define our transformation. We are free to start with $|\psi\rangle \to U^\dagger |\psi\rangle$. Then we'd also have to change the operator transformation to $\tilde{O} = U^\dagger O U$ to remain consistent.
There is more to be said about symmetry groups and how they act on a quantum system. One thing I want to point out however is that one can show that you can always find a representation of a symmetry group such that the action on states is given by unitary operators $U = e^{iA}$ where $A=A^\dagger$ is a hermitian matrix.
If $U$ is a symmetry of the Hamiltonian $H = U H U^\dagger$ (note that timedependent transformations require a special treatment if we are considering the Hamiltonian) we can rewrite this as $HU = UH$ or $[H, U] = 0$. This can be used to generate further eigenvalues of the Hamiltonian and classifying them according to their quantum numbers. Again it doesn't matter if we were to write $H = U^\dagger H U$ instead, we still arive at $[H, U] = 0$ which is really the key point.
