How to take trace ${\rm Tr}(\left |a\rangle \langle a|X|b\rangle \langle b \right|) $ in bra ket notation? Trace ${\rm Tr}(\left |a\rangle \langle a|X|b\rangle \langle b \right|) =$? Where $a$ and $b$ are two orthogonal state and $X$ is any operator.
 A: All you need is ${\rm tr}\{X\}=\sum_a \langle a|X|a\rangle$, where $|a\rangle$ is a compete set of orthonormal states.
A: It turns out that the cyclic property of the trace, i.e.,
$$
\operatorname{Tr}(ABC) = \operatorname{Tr}(CAB) = \operatorname{Tr}(BCA),
$$
holds when the matrices in question are not square.  In particular, interpreting the kets and bras as row and column vectors, respectively, one can write
$$
\operatorname{Tr}(|a\rangle\langle a|X|b\rangle\langle b|)
=
\operatorname{Tr}(\langle b|a\rangle\langle a|X|b\rangle)
=
\operatorname{Tr}(\delta_{ab}\langle a|X|b\rangle)
=\delta_{ab}\langle a|X|b\rangle,
$$
since the trace of a "number" is just the number (in the sense that it's a one-by-one matrix). So, if the states are orthogonal, then that trace is zero.
A: A less subtle alternative to using the cyclic property is "brute forcing" it, by plugging in the definition of the trace. It goes like this,
$$\begin{aligned}
{\rm Tr}(\left |a\rangle \langle a|X|b\rangle \langle b \right|) &= \sum_n\langle n | \big(|a\rangle \langle a|X|b\rangle \langle b| \big) |n\rangle \\
&= \sum_n\langle n |a\rangle \langle a|X|b\rangle \langle b |n\rangle\\
&= \langle a|X|b\rangle  \sum_n\langle n |a\rangle  \langle b |n\rangle\\
&= \langle a|X|b\rangle  \sum_n \langle b |n\rangle \langle n |a\rangle  \\
&= \langle a|X|b\rangle  \langle b| \bigg( \sum_n  |n\rangle \langle n| \bigg )|a\rangle  \\
&= \langle a|X|b\rangle  \langle b|\hat 1|a\rangle  \\
&= \langle a|X|b\rangle  \langle b|a\rangle  \\
{\rm Tr}(\left |a\rangle \langle a|X|b\rangle \langle b \right|) &= \langle a|X|b\rangle  \delta_{ba}  
\end{aligned}$$
where the $\{|n\rangle\}$ form an orthonormal set. The "trick" is to recognize the resolution of identity $\sum_n|n\rangle \langle n|=\hat 1 $ after rearanging the scalar products and the sum.
Another simplification that has not been mentioned yet, is also realizing that $\langle a|X|b\rangle=X_{ab}$ is just a complex number, and since the trace is a linear operation, we can pull this constant number out before evaluating the trace
$$\begin{aligned}
{\rm Tr}(\left |a\rangle \langle a|X|b\rangle \langle b \right|) &={\rm Tr}(\left |a\rangle X_{ab} \langle b \right|) \\
&=X_{ab}{\rm Tr}(\left |a\rangle  \langle b \right|) \\
&=X_{ab}\langle a|b\rangle \\
&=X_{ab}\delta_{ab}
\end{aligned}$$
