Let's say the bigger radius is $r_1$ and the smaller one $r_2$. The question really has to do with the direction of friction force. I assign the friction coefficient $\mu$, make the direction the force is acting as positive motion and write the sum of the forces in the horizontal direction, and sum of moments about the center:
$$ F-\mu\, m g\, {\rm sign}(v+r_1 \omega) = m \dot{v} \\
F r_2 - \mu\, m g\, r_1\, {\rm sign}(v+r_1 \omega) = I \dot{\omega} $$
where ${\rm sign}()$ returns -1
,0
or 1
depending on the sign of the argument, and $v$ is linear velocity of center, $\omega$ angular velocity of center, $m$ the total mass, and $I$ the mass moment of inertia at the center.
There are three posibilities:
- Yo-yo is rolling with the contact point velocity zero, $v+\omega r_1=0$ and motion
$$ v = \frac{F}{m} t \\ \omega = \frac{F r_2}{I} t $$
- Yo-yo is slipping with the contact point moving to the right, $v+\omega r_1>0$ and motion
$$ v = \frac{F-\mu\,m g}{m} t \\ \omega = \frac{F r_2-\mu\,m g\,r_1}{I} t $$
- Yo-yo has backspin with the contact point moving to the left, while the center moves to the right, $v+\omega r_1<0$ with motion
$$ v = \frac{F+\mu\,m g}{m} t \\ \omega = \frac{F r_2+\mu\,m g\,r_1}{I} t $$
Now I can check the conditions where is possibility can exist.
First if the friction is zero or less than $\mu = \frac{F (I+m r_1 r_2)}{m g (I+m r_1^2)} $ then we have the slipping condition since $$v+\omega r_1 = \frac{F-\mu\,m g}{m} t + r_1 \frac{F r_2-\mu\,m g\,r_1}{I} t > 0 $$
Otherwise if the friction is higher than $\mu = \frac{F (I+m r_1 r_2)}{m g (I+m r_1^2)} $ then we have an interesting situation. The problem is that $\mu$ is not constant, as the yo-yo goes through a slip-stick scenario.
The best I can do is treat friction as linear to slip velocity by some arbitrary coefficient (damping) $d$ such that $\mu = d v_c = d (v + \omega\,r_1) $. I can find the acceleration of the contact point by $\dot{v}_c = \dot{v} + \dot{\omega}\,r_1 $ which is used to find the equation of motion of the contact point:
$$ \dot{v}_c = \left(\frac{F-d\,m g\,v_c}{m}\right) + \left( \frac{F r_2 - d\, m g\,r_1\,v_c}{I}\right) r_1 \\
\dot{v}_c = \left(\frac{1}{m}+\frac{r_1 r_2}{I}\right) F - d\,g\,\left(1+\frac{m r_1^2}{I}\right) v_c \\
\dot{v}_c = a_0 - \beta v_c $$ where $a_0 = \left(\frac{1}{m}+\frac{r_1 r_2}{I}\right) F$ and $\beta = d\,g\,\left(1+\frac{m r_1^2}{I}\right)$. This differential equation has solution
$$ v_c = \frac{a_0}{\beta} \left(1-\boldsymbol{e}^{-\beta t}\right) \\ \dot{v}_c = a_0 \boldsymbol{e}^{-\beta t}$$
With that and the equations of motion the center of the yo-yo moves by
$$ \dot{v} = a_0 \left(\frac{I}{I+m r_1^2}\boldsymbol{e}^{-\beta t} + \frac{(r_1-r_2) I m r_1}
{(I+m r_1 r_2)(I+m r_1^2)}\right)$$
This is just a decreasing function. More interesting is the angular acceleration and velocity which both cross the zero line
$$ \dot{\omega} = a_0 \left(\frac{m r_1}{I+m r_1^2}\boldsymbol{e}^{-\beta t} -\frac{m I (r_1-r_2)}{(I+m r_1 r_2)(I+m r_1^2)}\right)$$ and $\omega = \int \dot{\omega}\,{\rm d} t$
$$ \omega = a_0 \left( \frac{m r_1}{\beta (I+m r_1^2)} \left(1-\boldsymbol{e}^{-\beta t}\right) - \frac{m I (r_1-r_2)}{(I+m r_1 r_2)(I+m r_1^2)} t \right) $$
Lastly the coefficient of friction varies by
$$ \mu = \frac{I a_0 \left(1-\boldsymbol{e}^{-\beta t}\right) }{g (I+m r_1^2)} $$
which starts from zero and stabilizes quickly at
$$ \mu = \frac{I+m r_1 r_2}{m g (I+m r_1^2)} F $$
The above can also be expressed in terms of the angular velocity as $\mu = \frac{r_2}{r_1} \frac{F}{m g} - \frac{I \dot{\omega}}{m g\,r_1} $.