The other answers here are in the spirit of what you can do, but allow me to elaborate a little more.
To understand if the trajectory of the movement under a potential $V $ is stable or not you have to understand what this stability means.
The most simple example is the harmonic oscillation- $V=-{1 \over 2}kx^2 $.In Newtonian mechanics, for a point of stability,you have that $F(r_0)=0 $ where $r_0 $ is the point of stability. That is equal(if the $\nabla \times \bar F$ is zero) to the curve of the potential. So we have to find whether the $$\nabla V(r_0)=0 $$ gives us minimum values for the potential $V(r_0)$ or maximum values. Then, in a diagram V=V(r) the maximum obviously gives you, for a little transposition dr, the body moving away from an unstable point of stability. In the other hand, a minimum will give for the same transposition, a movement inside that like-a-well potential unless more energy is given to the body than the topical minimum has. Here's a picture:
It is useful and maybe fundamental to see this diagram in phase space(space of velocity and coordinate). Here:
The problem with this method is that if the scalar function $V(r)$ is a function of many variables(more than one) the mathematics require a lot of work.
So we use the method of disturbance(note: maybe my translation to English here is wrong). That means to take the force $F$ and write the Taylor expansion around a point(like mentioned in another answer) at distance e:
$$F(x_0+e)=F(x_0) + \left({dF \over dx}\right)|_{x=x_0}e + {1 \over 2}\left({d^2 F \over dx^2}|_{x=x_0}\right)e^2 +... =0$$. From here we take a linear differential equation whose solution is the behaviour of the particle
around $x_0$.
And next, if we have a Lagrangian $L=T-V $ where T the kinetic energy, with the general coordinates q in relation with the Cartesian are not a function of time, we have:
$$T=\sum_{i=1}^n \sum_{j=1}^n a_{ij}(q_1 ... q_n)\dot q_i \dot q_j $$ and $$V=V(q_1 ... q_n) $$ the equations of Lagrange are: $$\sum (q_1 .. q_n) \ddot q_i + \sum \sum b_{ijk}(q_1 ... q_n)\dot q_i \dot q_j + {\partial V \over \partial {q_k}}=0 $$ and k=0,1,2..n
For equilibrium solution the terms with derivatives over time are zero and so: $$\left.{ \partial V \over \partial {q_k}} \right|_{q_0} =0 $$.
For a little movement around the equilibrium point we can then prove:
$$L=\left(\sum_{i=1}^n m_{ij}\ddot χ_i + b_{ij}χ_i\right)=0 $$
Finally, who do we study the stability of circular orbits?
We have, for the equilibrium points: $$F(r)+{L^2 \over mr^3}=0 --> {dV(r) \over dr}=0 $$
For stable trajectory $V(r=r_0)''>0 $ where $V''$ is the second derivative over r of the potential. from here we can prove that:$$V''|_{r_0}=-F'|_{r_0} +{3 \over r_0}{L^2 \over mr{_0}^3}>0 -->{F'|_{r_0} \over F|_{r_0} + }+ {3 \over r_0 }>0$$. That's the condition for stable trajectories.
I know that's a lot, but I hope it helps.