Confusion on symmetry and basis transformation Let {$|a_n\rangle$} and {$|b_n\rangle$} be two basis related by: $|b_n\rangle = \hat{U}|a_n\rangle \forall n$.
From my understanding then the unitary operator $\hat{U}$ only transforms the basis   {$|a_n\rangle$} into {$|b_n\rangle$} (just like in 2D geometry having a rotation operator which changes the basis $\hat{x},\hat{y}$ to $\hat{r},\hat{\theta}$).
If there is an operator $\hat{\Omega}$, then its representation in basis {$|b_n\rangle$}:
$$
\langle b_n|\Omega|b_m\rangle = \langle a_n| \hat{U}^\dagger\Omega\hat{U}|a_m\rangle
$$
$$\Omega \to \hat{U}^\dagger\Omega\hat{U}$$
On the other hand, consider the following unitary transformation:
$$|\psi\rangle = \Omega|\phi\rangle$$
$$\hat{U}|\psi\rangle = \hat{U}\Omega\hat{U}^\dagger\hat{U}|\phi\rangle$$
$$\Omega \to \hat{U}\Omega\hat{U}^\dagger$$
1)I am getting very confused by the difference between these, shouldn't the operator $\Omega$ transform in the same way?What is the difference between the two things I am doing?
 A: In the first example, you're taking $|b_i\rangle$ into $|a_i\rangle$, while in the second, you're doing the opposite. Explicitly, if 
$$|\psi\rangle = \sum_n b_n|b_n\rangle$$
then 
$$\hat U|\psi\rangle = \sum_n b_n\hat U|b_n\rangle.$$
But we don't know anything about $U|b_n\rangle$! To go from $|b_i\rangle$ into $|a_i\rangle$, we what we need is $\hat U^\dagger$. Then the calculation will come out as $\Omega \mapsto \hat U^\dagger \Omega \hat U$.
A: I suppose I'll formally write this up since there seems to still be some confusion. Let's firmly establish that our $U$ is a transformation from $a$ to $b$, that has it's representation in the $a$ basis as 
$$\langle a_i |U|a_j\rangle = \langle a_i|b_j\rangle$$
Let's look at how the representation of $|\psi\rangle$ in the $a$ basis transforms when we go to the $b$ basis:
$$|\psi\rangle = \sum_{j}|b_j\rangle\langle b_j|\psi\rangle=\sum_{j}\sum_{i}|b_j\rangle\langle b_j|a_i\rangle\langle a_i|\psi\rangle$$
Now pick out the $k$'th component of $b$
$$\langle b_k|\psi\rangle = \sum_{j}\sum_{i}\delta_{kj}\langle b_j|a_i\rangle\langle a_i|\psi\rangle = \sum_{i}\langle b_k|a_i\rangle\langle a_i|\psi\rangle \\ = \sum_{i}(\langle a_i|b_k\rangle)^{\dagger}\langle a_i|\psi\rangle = \sum_{i}(\langle a_i|U|a_k\rangle)^{\dagger}\langle a_i|\psi\rangle.$$
Letting subscripts denote the basis (i.e. $|\psi\rangle_a \equiv \langle\vec{a}|\psi\rangle$, and likewise for $b$), we see that this is telling us $|\psi\rangle_b = U^{\dagger}|\psi\rangle_a$. Now, from $U|a_i\rangle=|b_i\rangle$ we know that $\Omega_b = U^{\dagger}\Omega_a U$, so lets check that everything is consistent with your $|\psi\rangle = \Omega|\phi\rangle$ when we hit it with $U^{\dagger}$. Keeping subscripts to denote the basis for absolute clarity:
$$|\psi\rangle_a = \Omega_a|\phi\rangle_a \to U^{\dagger}|\psi\rangle_a = U^{\dagger}\Omega_a|\phi\rangle_a$$
Looking at each side individually, we have
$$U^{\dagger}|\psi\rangle_a = |\psi\rangle_b \\ U^{\dagger}\Omega_a|\phi\rangle_a =U^{\dagger}\Omega_a U U^{\dagger}|\phi\rangle_a = \Omega_b |\phi\rangle_b,$$
which shows us that everything is nice and consistent:
$$ |\psi\rangle_a = \Omega_a|\phi\rangle_a \xrightarrow{U^{\dagger}} |\psi\rangle_b = \Omega_b|\phi\rangle_b $$
A: $\let\Om=\Omega \let\dag=\dagger \def\ket#1{|#1\rangle} \def\bra#1{\langle#1|} \def\braket#1#2{\langle#1|#2\rangle} \def\mxelm#1#2#3{\bra#1\,#2\,\ket#3}$
As you'll see, I'll make some change of notations I can't explain for brevity. In your first definition you're asking for an operator $\Om'$ which in $a$-rep has the same matrix elements $\Om$ has in $b$-rep:
$$\mxelm{{a,m}}{\Om'}{{a,n}} = \mxelm{{b,m}}\Om{{b,n}}$$
$$\Om' = U^\dag\,\Om\,U.$$
Afterwards you assume some operator $\Om$ sends ket $\xi$ into $\eta$:
$$\ket\eta = \Om\,\ket\xi.$$ 
Then you define $\ket{\xi'}$, $\ket{\eta'}$ such that they have in $b$-rep the same components $\ket\xi$, $\ket\eta$ have in $a$-rep:
$$\braket{b,n}{\xi'} = \braket{a,n}\xi$$
$$\mxelm{{a,n}}{U^\dag}{{\xi'}} = \braket{a,n}\xi$$
$$\ket{\xi'} = U\,\ket\xi \qquad \ket{\eta'} = U\,\ket\eta.$$
Lastly you look for $\Om'$ sending $\ket{\xi'}$ into $\ket{\eta'}$:
$$\Om'\,U\,\ket\xi = U\,\ket\eta = U\,\Om\,\ket\xi.$$
If this is to hold for all $\ket\xi$ then
$$\Om' = U\,\Om\,U^\dag.$$
It should be clear that in your former part you transformed an
operator, whereas in the latter you transformed kets under the
opposite requirement.
