Action of unitary operator on operator: two different definitions? Defnition 1:
Ref: Section 2.4.9, Quantum Mechanics: Concepts and Applications
By Nouredine Zettili
Ket $|\psi \rangle $ transform as $\underline{|\psi' \rangle = \hat{U} |\psi \rangle  }$. Given this, the transformation of any operator can be derived as follows:
    $$\hat{A}|\psi \rangle = |\phi \rangle \rightarrow \hat{A}' |\psi'\rangle =|\phi'\rangle.
$$ Now, $ \hat{A}' |\psi'\rangle =|\phi'\rangle$ can be written as
     $$\hat{A'}\hat{U}|\psi\rangle = \hat{U}|\phi\rangle = \hat{U}\hat{A}|\psi\rangle
$$
which leads to 
    $$\hat{A}' \hat{U} = \hat{U}\hat{A} \rightarrow \underline{\hat{A}' = \hat{U}\hat{A}\hat{U}^{\dagger}}
$$
Defnition 2:
In Schroedinger picture, $|\psi \rangle$ transforms as
$\underline{|\psi\rangle  \rightarrow \hat{U}|\psi \rangle }$.
   Any other operator $\hat{A}$ in Hesisenberg picture evolve as
$$\underline{\hat{A} \rightarrow \hat{U}^{\dagger} A \hat{U}}
$$
which is clearly different than the one obtained in definition 1. 
Why there is a difference?
Do the two definitions represents different scenario? 
 A: edit: I've given a more clear summary of my response below starting at "Two Things".
Commonly I've seen things defined as such. The action of a unitary operator on kets is defined as:
$$
|\psi'\rangle = U|\psi\rangle
$$
And we say $|\psi'\rangle$ is the result of acting $U$ on $|\psi\rangle$. 
Let $A'$ be the result of acting $U$ on $A$. I haven't yet defined what this means but I will shortly. We desire that
$$
\tag{*}
\langle \psi'|A'|\psi'\rangle = \langle \psi|A|\psi\rangle
$$
We expand
$$
\langle \psi'|A'|\psi'\rangle = \langle \psi|U^{\dagger}A'U|\psi\rangle
$$
So we see that (according to $(*)$) we have
$$
U^{\dagger}A'U = A
$$
So that
$$
A' = UAU^{\dagger}
$$
We then say that $A'$ is the result of acting $U$ on $A$.
I'm not sure where you discrepancy for the Heisenberg picture approach comes from but it might be the following. 
(considering a time-independent Hamiltonian) There is an operator
$U_H(t) = e^{-i\frac{H}{\hbar} t}$ Where $H$ is the Hamiltonian of the system. Time evolution in the Schrodinger picture can be calculated as
$$
|\psi(t)\rangle = U_H(t)|\psi_0\rangle
$$
That is, time evolution can be described as the action of $U_H(t)$ on $|\psi_0\rangle$.
It is tempting then to say that time-evolution in the Heisenberg pictured is effected by acting $U_H(t)$ on an operator $A$. However this is incorrect. Using the language I have given above, time-evolution of operators is effected by acting $U_H(t)^{\dagger}$ on operators.
That is
$$
A(t) = U_H(t)^{\dagger}A_0 U_H(t)
$$
Two Things:
1) Action of a unitary on an operator.
As above we have that action of $U$ on $|\psi\rangle$ is defined by
$$
|\psi'\rangle = U|\psi\rangle
$$
We desire 
$$
\langle\psi'|A'|\psi'\rangle = \langle \psi|U^{\dagger}A'U|\psi\rangle = \langle \psi|A|\psi\rangle
$$
Which constrains 
$$
U^{\dagger}A'U = A
$$
$$
A' = UAU^{\dagger}
$$
2) Heisenberg time evolution consider the time-evolving expectation value:
Let $U$ be the time-evolution operator which is defined by $U=e^{-i\frac{H}{\hbar} t}$ where $H$ is the Hamiltonian of the system. (Caution, this definition only works for time-independent Hamiltonians but can be generalized.)
\begin{align}
\langle A\rangle = \langle \psi_0| U^{\dagger}(t)A_0 U(t)|\psi_0\rangle
\end{align}
Define the Schrodinger operators and kets by
\begin{align}
|\psi_S\rangle &= U|\psi_0\rangle\\
A_S &= A_0
\end{align}
With this definition we see
$$
\langle A \rangle = \langle \psi_S |A_S|\psi_S\rangle
$$
Alternative we can define the Heisenberg operators and kets by
\begin{align}
|\psi_H\rangle &= |\psi_0\rangle\\
A_H &= U^{\dagger}A_0U
\end{align}
In which case we see
$$
\langle A \rangle = \langle \psi_H|A_H|\psi_H\rangle
$$
Summary
So we see that the action of a unitary operator, $U$ on $A$ is given by
$$
A' = UAU^{\dagger}
$$
and this definition arises from the constraint that $\langle \psi | A |\psi\rangle = \langle \psi' |A'|\psi'\rangle$
Next, we see that the Heisenberg version of an operator, $A_H$ is given by
$$
A_H = U^{\dagger} A U
$$
And this constraint arises from the constraint that $\langle \psi_S | A_S |\psi_S \rangle = \langle \psi_H|A_H|\psi_H\rangle = \langle \psi_0|U^{\dagger}A_0U|\psi_0\rangle$.
We see that $A_H$ is eqiuvalent to the action of $U^{\dagger}$ on $A_0$.
A: These are two different types of transformation, which indeed apply in different scenarios.  The first is a rotation (or change of basis).  In this case, you want the action of the transformed operator $A'$ on the transformed state $|\psi'\rangle$, in the new basis, to be the same as the action of the original operator $A$ on the original state $|\psi\rangle$, in the old basis.  You can see that this works out, as you noted, by
$$A'|\psi'\rangle=UAU^{\dagger}U|\psi\rangle=UA|\psi\rangle.$$
The second "definition" you describe is a time evolution. In the Schrodinger picture, the state evolves, and in the Heisenberg picture, the operator evolves, but in either case the action of the operator on the state should change with time, i.e. after application of $\hat{U}$. We can see that this works out properly: if $A$ is the Schrodinger picture operator and $|\psi\rangle$ is the Schrodinger picture state, then in the Schrodinger picture
$$|\phi\rangle_{\text{Schr}}=A\hat{U}|\psi\rangle$$
is the action of $A$ on the time-evolved state $\hat{U}|\psi\rangle$. In the Heisenberg picture, the action of the time-evolved operator $\hat{U}^{\dagger}A\hat{U}$ on the (non-evolving) state $|\psi\rangle$ is
$$|\phi\rangle_{\text{Heis}}=\hat{U}^{\dagger}A\hat{U}|\psi\rangle,$$
so we can see that the $|\phi\rangle$s in the two pictures are related as they should be:
$$|\phi\rangle_{\text{Heis}}=\hat{U}|\phi\rangle_{\text{Schr}}.$$
Note that neither of these is, in general, equal to the action of $A$ on the non-time-evolved state $|\psi\rangle$.
A: You can use either definition. All physical quantities are expectation values of an operator. In the Schrodinger picture the time evolution of the expectation value $\langle \psi | \hat A |\psi\rangle$ comes from the wavefunction. In the Heisenberg picture it comes from the time evolution of $\hat A$. 
Schrodinger: The observable is constant in time.
\begin{gather}
|\psi(t)\rangle = U(t, t_0)|\psi(t_0)\rangle \\
\hat A \neq \hat A(t) \\
\langle \hat A \rangle = \langle \psi(t)|\hat A|\psi(t)\rangle
\end{gather}
Heisenberg: The basis is constant in time.
\begin{gather}
|\psi(t)\rangle =\psi(t_0)\rangle \\
\hat A = \hat A(t) \\
\langle \hat A \rangle = \langle \psi|\hat A(t)|\psi\rangle
\end{gather}
