Stability of circular orbit in attractive inverse cube central force field Considering a motion of a body under an attractive inverse cube central force,
$\textbf{F}(\textbf{r}) = -\frac{k}{r^3} \hspace{1mm}\hat{\textbf{r}}$ with $k>0$.
Is it possible for a body to move in an stable circular orbit? Since the derivation of the effective potential
$U_{eff}(r) = \frac{l^2}{2mr^2}+U(r)$ 
(where $l$ is the angular momentum)
has to be $0$ for a circular orbit, the only solution would be that $k = \frac{l^2}{m}$. But that would lead to an effective potential $U_{eff}(r) = 0$ for any $r$ (except $r = 0$). Is this a valid solution? 
 A: For stable orbit, we need to have $d^2V_{eff}/dr^2>0$ at $r=r_0$ and we need to find $r_0$ from the solution of $dV_{eff}/dr=0$ 
We can find the potential by $$V=-\int Fdr$$
Hence $$V=-\frac {k} {2r^2}$$ so $$V_{eff}=\frac {l^2} {2mr^2}-\frac {k} {2r^2}$$
and at $r=r_0$, $dV_{eff}/dr=0$ hence
$$dV_{eff}/dr=\frac {-l^2} {mr_0^{3}}+\frac {k} {r_0^3}=0$$
so we have,
$$l^2=mk$$
Now we need to find $d^2V_{eff}/dr^2$ at $r=r_0$
$$d^2V_{eff}/dr^2=\frac {3l^2} {mr_0^{4}}-\frac {3k} {r_0^4}$$
Using the above relationship we find that, 
$$d^2V_{eff}/dr^2=\frac {3k} {r_0^{4}}-\frac {3k} {r_0^4}=0$$ which is exactly zero. So there cannot be any stable circular orbit.
A: The possible trajectories of a particle subject to an inverse-cube force $F = - k/r^3$ can actually be derived exactly;  they are known as Cotes's spirals.  Depending on the relative values of the particle's angular momentum $\ell$, its mass $m$, and the constant of proportionality $k$ in the force law, they take on the form
$$
r(\theta) = \begin{cases} (A \cos C \theta + B \sin C \theta)^{-1} & km < \ell^2 \\ (A \cosh C \theta + B \sinh C \theta)^{-1} & km > \ell^2 \\ (A + B \theta)^{-1} & km = \ell^2 \end{cases}
$$
where
$$
C = \sqrt{\left| \frac{ m k}{\ell^2} - 1 \right|}
$$
and $A$ and $B$ are determined by the initial conditions of the trajectory.
It is not hard to see that almost all of these functions will either have $r \to \infty$ for some finite value of $\theta$, or $r \to 0$ as $\theta \to\infty$, or both.  The only case in which $r$ is bounded and does not reach $r = 0$ is when $km = \ell^2$ and $B = 0$, which corresponds to circular motion. Most perturbations from this trajectory will either involve changing $\ell$ (if the particle is given an extra tangential "push"), or changing $B$ (if the particle is given a purely radial push, since now we must have $dr/d\theta = 0$.)  Thus, most perturbations will lead to the particle either spiraling in to $r = 0$ or flying out to $r \to \infty$.
Thinking about this in terms of the effective potential:  to have a circular orbit in an inverse-cube field, you must have $\ell = \sqrt{km}$.  A perturbation will either change $\ell$ or leave $\ell$ the same.  If $\ell$ is changed from its initial value, then the effective potential becomes $U_\text{eff}(r) = Q/r^2$ for some value of $Q$, which has no maxima or minima;  the radial motion must either go to $0$ or $\infty$.  If $\ell$ is unchanged, then we must have $dr/dt = 0$, and the effective 1-D problem is that of a particle moving with some initial velocity in a potential $U_\text{eff} = 0$.  Again, this radial motion must either go to $0$ or $\infty$.
A: for a circle  motion die circle radius $r(t)$ must be constant.
we can calculate the EOM's with Euler-Lagrange method.
kinetic energy:
$$ T=\frac{1}{2}\,m\left(\dot{r}^2+r^2\,\dot{\phi}^2\right)$$
potential energy
$$U=\frac{L^2}{2\,m\,r^2}+ U(r)$$
where $L$ is the angular momentum and
$U(r)$  the unknown potential for a  circle  motion
The equation of motions are:
$$\ddot{r}=r\,\dot{\phi}^2+\frac{L^2}{m^2\,r^3}-\frac{1}{m}\frac{d\,U}{d\,r}\tag 1$$
$$m\,r\left(\ddot{\phi}\,r+2\,\dot{r}\,\dot{\phi}\right)=0\quad \Rightarrow$$ 
$$m\,\frac{d}{dt}\left(r^2\,\dot{\phi}\right)=0\tag 2$$
we obtain from  equation (2)  that $\dot{\phi}=h/r^2\quad$ with $h=L/m$
we put this result in equation (1) and get:
$$\ddot{r}=\frac{2\,L^2}{m^2\,r^3}\,-\frac{1}{m}\,\frac{d\,U}{d\,r}\tag 3$$
for a circle motion $r(t)=const\quad \Rightarrow\quad \ddot{r}=0$  ;
we solve equation (3) for $\frac{d\,U}{d\,r}$
$$\boxed{F(r)=\frac{d\,U}{d\,r}=\frac{2\,L^2}{m}\frac{1}{r^3}}$$
A: In case this helps anyone in the future. Defining Veff essentially turns this into a 1d problem with only r as the dynamical variable. If we want circular orbit you can either  (do explicit calculation as Layla's answer does) or just see that Veff = const × 1/r^2 for inverse-cubic force law. Now, this function can be either monotonically increasing (const<0) or decreasing (const>0) except when the const =0. For monotonic functions you don't get any stationary points of Veff so no circular orbit is possible. Only when const = 0 => Veff = 0, you can get a circular orbit.
Talking about stability of circular orbit: As Veff is just zero for all r, so it's like a 1d question of a ball on a plane surface. If you give it a push it will just keep going. If the push is inwards it will collide with force center, or if the push is outward then it will move put to infinity. In 2D, our particle will just trace out a spiral inwards or outwards respectively.
