Rotating frame transformation I have the following Hamiltonian 
$$\hat{H} = \omega_0 \hat{S}_z + 2\Omega\cos(\omega t)\hat{S}_x$$
The author says the in the rotating frame of reference the Hamiltonian becomes
$$\hat{H} = (\omega_0-\omega)\hat{S}_z + \Omega \hat{S}_x$$
What kind of transformation leads from first Hamiltonian to the second and what happens with the wavefunction?
 A: Literally the result is wrong. However it seems to be correct if taking the expectation value over a period of time $T= 2\pi/\omega$, provided my computations are correct.
I assume the system is completely described in the space of spin. Consider the state $\psi'$ in the rotating frame compared with the analog $\psi$ in the fixed reference frame. I assume the rotation happens around the $z$ axis with pulsation $\omega$ and $\hbar=1$.
$$\psi'(t) = e^{i\omega t S_z} \psi(t)\:. \tag{1}$$
NB. I am assuming here that physics in the rotating frame is the same as in a class of inertial reference frames, a different reference frame at every instant of time and connected with the initial inertial reference  by means of a class of rotations, different at every instant, parametrized by $t$.
With this quite strong assumption, in writing (1), I exploited  the fact that spatial rotations (in a given reference frame) are generated by the total angular momentum operator, here the spin only.
Therefore 
$$ i\frac{d}{dt}\psi'(t) = i\frac{d}{dt}e^{i\omega t S_z} \psi(t) =  -\omega S_z \psi(t) + e^{i\omega t S_z}  i\frac{d}{dt} \psi(t)=   -\omega S_z \psi'(t) + e^{i\omega t S_z} H(t) \psi(t) =  -\omega S_z \psi'(t) + e^{i\omega t S_z} H(t)e^{-i\omega t S_z} \psi'(t) \:.$$
Summing up,
$$ i\frac{d}{dt}\psi'(t) =  \left(\omega S_z + e^{i\omega t S_z} H(t)e^{-i\omega t S_z}\right) \psi'(t) \:.$$
In summary, assuming that $\psi'$ satisfies a Scroedinger equation, its Hamiltonian (the one in the rotating frame) must be
$$H'(t) = -\omega S_z + e^{i\omega t S_z} H(t)e^{-i\omega t S_z}\:.$$
In other words, using the known form of $H(t)$, we have
$$H'(t) = (\omega-\omega_0) S_z + 2\Omega\cos(\omega t) e^{i\omega t S_z} S_x e^{-i\omega t S_z}\:.$$
That is 
$$H'(t) = (\omega-\omega_0) S_z + 2\Omega\cos(\omega t)\left(\cos(\omega t) S_x - \sin(\omega t) S_y\right)\:.$$
This is the correct Hamiltonian (please check signs).
Now, an effective Hamiltonian, if $T = 2\pi/\omega$ is very short with respect to a reference time proper of experimental observations, we can use the effective Hamiltonian
$$H'(t)_{eff} = \frac{1}{T}\int_{0}^T H'(t) dt\:.$$
Since $(\omega-\omega_0) S_z$ does not depend on time and $\cos(\omega t)\sin(\omega t) = \frac{1}{2}\sin(2\omega t)$ has zero integral over $[0,T]$ for $T=2\pi/\omega$, it remains
$$H'(t)_{eff} = (\omega-\omega_0) S_z + 2\Omega \frac{1}{T}\int_0^T\cos^2(2\pi t/T) dt S_x \:.$$
Since 
$$\frac{1}{T}\int_0^T\cos^2(2\pi t/T) dt =\frac{1}{2}$$
we find
$$H'(t)_{eff} = (\omega-\omega_0) S_z + \Omega  S_x \:.$$
A: If we assume this Hamiltonian to describe a 2-state system, we can describe the state of the system using a Bloch spehere.
This Hamiltonian then describes a system rotating around the z-axis with angular velocity $\omega_0$. To this system an oscillating signal with frequency $\omega$ is applied. This signal rotatates the system back and forth around the x-axis. The maximum angular velocity with which it rotates around the x-axis is $2\Omega$, while the frequency with which it oscillates back and forth is $\omega$
A requirement for the rotating frame approximation to be used is that the angular velocity around the x-axis, $2\Omega$, is much smaller than $\omega_0$. This means that the system revolves many times around the z-axis before having revolved once around the x-axis. Effectively this means that there will be almost no rotation around the x-axis, because each time the system revolves an angle $\pi$ around the z-axis, the effect of the x-rotation will be reversed. 
This cancellation is true unless the the x rotation switches sign with the same frequency as the rotation frequency around the z-axis, thus when $\omega \approx \omega_0$. If that case the rotations around the x-axis will add up instead of cancel.
Up till now all rotations have been defined in the lab frame. It is often easier to define a frame that rotates along with the system around the z-axis with a frequency $\omega_0$. In this frame, the system obeys your second Hamiltonian. When $\omega \approx \omega_0$ there is almost no rotation around the z-axis and under influence of the external signal the system is rotated around the (new rotating) x-axis with angular velocity $\Omega$.
For a more mathematical approach see wikipedia
