Forms of transformation Suppose $O$ is an object to be transformed, and $S$ is the transformation operator. Sometime the transformation is in the form
\begin{equation}
O \rightarrow SO. \tag{1}
\end{equation}
But sometime the transformation is in the form
\begin{equation}
O \rightarrow SOS^{-1}.\tag{2}
\end{equation}
I am confused. I know that there is some difference between these two cases. I just don't know what is the difference? What kind of objects transform in the first way, and what kind of objects transform in the second way? Is there any rule?
 A: This is a question where the bra-ket notation is truly useful.  When $O$ is a column vector, think of it as a ket $\vert O\rangle$.  Then the transformation matrix $S$ acts in the usual way
$$
\vert O’\rangle=S\vert O\rangle\, . \tag{1}
$$
Now think of $O$ as an operator, i.e an object of the form $\vert m\rangle\langle n\vert$ that takes one vector into another, where $\vert m\rangle$ is a column vector but $\langle n\vert$ is a row vector.
Applying (1) to $\vert m\rangle$ and $\langle n\vert $ gives
\begin{align}
\vert m‘\rangle &= S\vert m\rangle\, ,\\
\langle n’\vert&= \langle n\vert S^{-1}
\end{align}
so that the operator $O$ now transforms as 
$$
O’= \vert m’\rangle\langle n’\vert = S\vert m\rangle\langle n\vert S^{-1}
=S O S^{-1}\, .
$$
This is not a formal proof but it shows (somewhat intuitively) that when the operator $O$ can be represented as a matrix, you can think of $S$ as required to transform the rows of $O$ and $S^{-1}$ as required to transform the columns of $O$.
A: Let me try a concrete example - this might not satisfy you, but maybe it will help you clarify. This is basically paraphrasing from Srednicki's Quantum Field Theory, chapter 2.
A Lorentz transformation is a coordinate transformation that acts like
$$x'^a=\Lambda^a_bx^b,$$
and preserves the spacetime interval. (This is an example of something which transforms like the first case you mention, $O\to SO$). The Lorentz transformations form a Lie group.
An infinitesimal Lorentz transformation can be written 
$$\Lambda^a_b=\delta^a_b+\delta\omega^a_b.$$
This is a particular case of a representation of the Lorentz group as a matrix $\Lambda\to\Lambda^a_b$. But if we're encoding symmetries in quantum theory, we might need other representations, and specifically, if we want probabilities to be preserved we need them to be unitary. Assume there is such a representation, $U(\Lambda)$. Since $\Lambda$ forms a group, this representation must satisfy
$$U(\Lambda'\Lambda)=U(\Lambda')U(\Lambda).$$
The infinitesimal transformation must look something like this:
$$U(1+\delta\omega)=1+\delta\omega_{ab}M^{ab},$$
where $M^{ab}$ are called the generators (I think of the Lie algebra now, but maybe a mathematician should correct me). Anyway, the following expression must be valid,
$$U(\Lambda^{-1}\Lambda'\Lambda)=U(\Lambda)^{-1}U(\Lambda')U(\Lambda)$$
and if you put in the infinitesimal transformation for $\Lambda'$, you get the following condition:
$$U(\Lambda)^{-1}M^{ab}U(\Lambda)=\Lambda^a_c\Lambda^b_cM^{cd}$$
so the objects $M^{ab}$ transform like your second case,
$$M\to U^{-1}MU.$$
So, here is one possible contextual remark we might be able to make: In many (some?) situations, a symmetry encoded as a Lie group transforms as your case 1, whereas the Lie algebra transforms as your case 2.
A: (1) is Lorentz transformation, while (2) is similarity transformation. 
Lorentz transformation includes rotation and boost. Similarity transformation is performed upon a square matrix that leaves invariant its characteristic polynomial, trace, and determinant. The transformed matrix is similar to the original matrix in the sense that they represent the same linear map under two different bases. The matrix $S$ in (2) is the change-of-basis matrix. In quantum mechanics and quantum field theory, similarity transformation is mostly used to diagonalize a matrix and find out its eigenvalues. See Wikipedia for the derivation of the transformation form (2).
