The argument here is to relate the phase space of self-sustained oscillator with the system of bead in a ring in viscous fluid.
Mathematically, the force field in this case come from the driving force $\epsilon \cos(\omega t + \bar{\phi_e})$. In the rotating frame of $\omega$, the driving force will become $\epsilon(e^{i \bar{\phi_e}}+e^{-i(2\omega t+ \bar{\phi_e})})$, the second term can be dropped if the detunning is small $2\omega \gg\Delta=|\omega_0 - \omega|$. Also, the original dynamic $\phi(t)$ will become $A e^{i((\omega_0 -\omega)t +\phi_0)}=A e^{i(\Delta t +\phi_0)}$. With zero detuning $\Delta=0$, the phase point doesn't change with time, which is what you expect when you travel along the same rotating frame as the particle.
What does potential mean?(I know that potential function defines an invariance most of the times, but how it is defnied for oscillators case?
If you don't like how the figure draw, you can just rotate the figure until the force vector pointing downward. Then it is exact a bead in a ring in viscous fluid under gravity and you should be able to understand most concept immediately.
For any conservative force field $F(r)$, potential energy $V(r)$ is always well defined by the integral $V(r)=-\int F(r)\cdot dr$. The force here is constant (so it is conservative). Its magnitude is given by $\epsilon$, and the direction depends on how it drive the system.
What are big parallel vectors and the small ones in figure 3.3(a) representing?
In Fig. 3.3, the big vectors is the "force vector" in phase space, small one is its parallel component to the ring.
For non-zero detuning, say $\Delta=|\omega_0 -\omega|>0$, you would expect the particle move counter-clockwise because it rotates faster than your rotating frame $\omega$. This create effective force as shown in Fig. 3.4. Its magnitude is constant, and its direction is always parallel to the ring as expected. Fig. 3.4 explains how these two force can balance out and create two points of equilibrium, or no point of equilibrium.
Why they phase locking take place in ϕ0
For a bead in a ring in viscous fluid under gravity, you will expect that the bead will move to the lowest point, which is $\phi_0$.
Why point 1 is stable and point 2 is not? I imagine stability means the point with minimum level of potential. I hope the answer to this question becomes clear if I get an answer for the first question.
The definition of stability at a point $r(t)$ is that the system will eventual go back to $r(t)$ for any small perturbation $\delta r$ as $t \to \infty$. The definition of equilibrium is that there is no net force.
It is obvious that both point 1 and 2 are equilibrium because there is no force along the ring. However, a small perturbation on the highest point will drive the bead to the lowest point 1, while perturbation on 1 will only drive it back to point 1. That is why point 1 is called stable equilibrium, but point 2 is called unstable equilibrium.
This picture can be somehow understood by the dynamics equation:
$$\ddot{x}+\omega_0^2x = f(x,\dot{x}) + \epsilon p(t)$$
The $f(x,\dot{x})$ is the internal dynamic and $\epsilon p(t)$ is the external driving force. To understand the basic idea of limit cycle, you need to know the simplest example: damped and driven harmonic oscillator.
I guess here $f(x,\dot{x}) = -\gamma \dot{x} + \tilde{A} \cos(\omega_0 t + \tilde{\phi}_0)$ in author's mind for the self-sustained oscillator right at the resonance frequency $\omega_0$ of the oscillator. Also external force should be $p(x)=\cos(\omega t + \bar{\phi_e})$.
