# How to take trace ${\rm Tr}(\left |a\rangle \langle a|X|b\rangle \langle b \right|)$ in bra ket notation?

Trace $${\rm Tr}(\left |a\rangle \langle a|X|b\rangle \langle b \right|) =$$? Where $$a$$ and $$b$$ are two orthogonal state and $$X$$ is any operator.

All you need is $${\rm tr}\{X\}=\sum_a \langle a|X|a\rangle$$, where $$|a\rangle$$ is a compete set of orthonormal states.
It turns out that the cyclic property of the trace, i.e., $$\operatorname{Tr}(ABC) = \operatorname{Tr}(CAB) = \operatorname{Tr}(BCA),$$ holds when the matrices in question are not square. In particular, interpreting the kets and bras as row and column vectors, respectively, one can write $$\operatorname{Tr}(|a\rangle\langle a|X|b\rangle\langle b|) = \operatorname{Tr}(\langle b|a\rangle\langle a|X|b\rangle) = \operatorname{Tr}(\delta_{ab}\langle a|X|b\rangle) =\delta_{ab}\langle a|X|b\rangle,$$ since the trace of a "number" is just the number (in the sense that it's a one-by-one matrix). So, if the states are orthogonal, then that trace is zero.
A less subtle alternative to using the cyclic property is "brute forcing" it, by plugging in the definition of the trace. It goes like this, \begin{aligned} {\rm Tr}(\left |a\rangle \langle a|X|b\rangle \langle b \right|) &= \sum_n\langle n | \big(|a\rangle \langle a|X|b\rangle \langle b| \big) |n\rangle \\ &= \sum_n\langle n |a\rangle \langle a|X|b\rangle \langle b |n\rangle\\ &= \langle a|X|b\rangle \sum_n\langle n |a\rangle \langle b |n\rangle\\ &= \langle a|X|b\rangle \sum_n \langle b |n\rangle \langle n |a\rangle \\ &= \langle a|X|b\rangle \langle b| \bigg( \sum_n |n\rangle \langle n| \bigg )|a\rangle \\ &= \langle a|X|b\rangle \langle b|\hat 1|a\rangle \\ &= \langle a|X|b\rangle \langle b|a\rangle \\ {\rm Tr}(\left |a\rangle \langle a|X|b\rangle \langle b \right|) &= \langle a|X|b\rangle \delta_{ba} \end{aligned} where the $$\{|n\rangle\}$$ form an orthonormal set. The "trick" is to recognize the resolution of identity $$\sum_n|n\rangle \langle n|=\hat 1$$ after rearanging the scalar products and the sum.
Another simplification that has not been mentioned yet, is also realizing that $$\langle a|X|b\rangle=X_{ab}$$ is just a complex number, and since the trace is a linear operation, we can pull this constant number out before evaluating the trace \begin{aligned} {\rm Tr}(\left |a\rangle \langle a|X|b\rangle \langle b \right|) &={\rm Tr}(\left |a\rangle X_{ab} \langle b \right|) \\ &=X_{ab}{\rm Tr}(\left |a\rangle \langle b \right|) \\ &=X_{ab}\langle a|b\rangle \\ &=X_{ab}\delta_{ab} \end{aligned}