How to apply anti-unitary symmetry operators? It is know that the symmetry operators can be applied to operators like
$$
\hat{O} \stackrel{g}{\rightarrow} \widehat{g O}
$$
demand the matrix element to be invariant under symmetry, we have
$$
\langle g \psi|\widehat{g O}| g \phi\rangle=\langle\psi|\hat{O}| \phi\rangle
$$
if the symmetry operator is unitary, we get
$$
\langle \psi| g^{-1}\widehat{g O} g| \phi\rangle=\langle\psi|\hat{O}| \phi\rangle
$$
so the unitary symmetry operator acting on operators can be represented by
$$
\widehat{g O} = g \hat{O} g^{-1}
$$
$\textbf{However}$, the anti-unitary symmetry operators have problems in the third equation in that they read
$$
\overline{\langle \psi| g^{-1}\widehat{g O} g| \phi\rangle}= \langle\psi|\hat{O}| \phi\rangle
$$
which can not lead to the fourth equation.
On the other hand, every textbook says that the time reversal symmetry operator (as an anti-unitary operator) acting on the operators can be represented by
$$
\widehat{T O} = T \hat{O} T^{-1}
$$
Now I wonder in what sense does this argument go wrong and how to derive the anti-unitary symmetry action on the operators.
 A: You forgot the crucial fact that the observables are selfadjoint and that the relevant matrix elements have the same entries. Only those matrix elements  are physically measurable since they are expectation values / probabilities (if the observable $A$ beloe is an orthogonal projector).
Suppose that $U$ is antiunitary and $A$ selfadjoint, then
$$\langle \psi| U A U^{-1} \psi \rangle = \overline{\langle U^{-1}\psi| U^{-1}(U A U^{-1} \psi) \rangle} = \overline{\langle U^{-1}\psi|  A U^{-1} \psi \rangle}$$
Now notice that $A=A^*$ so that $\langle \phi|A\phi\rangle = \overline{\langle \phi|A\phi\rangle}$. Using this property above, we find
$$\langle \psi| U A U^{-1} \psi \rangle = \langle U^{-1}\psi|  A U^{-1} \psi \rangle\:.$$
In other words, changing the state $\psi \to U^{-1}\psi$ is equivalent to changing the observables $A \to UAU^{-1}$, exactly as in the unitary case.
The result extends to mixed states.
$$tr(\rho UAU^{-1}) = tr(U^{-1}\rho U A)\:.$$
This identity is trivial when $U$ is unitary by taking advantage of the cyclic property of the trace. If $U$ is antiunitary, the proof can be performed by explicitly computing the traces on both sides and exploiting an argument as in the case above of a pure state.
