Considering a potential energy of $U$, and a displacement of $x$, the force is given by
$F=-\frac{\partial U}{\partial x}$.
Since equilibrium is defined as the point at which $F=0$, we can express this as $\frac{\partial U}{\partial x}=0$. This is clear to see on the following graph;
It is also clear that some equilibria are stable and some are not; given a small displacement at $x_2$ the system will return to equilibrium, whereas this would not happen at $x_3$. Hence, we can say that for $\frac{\partial ^2U}{\partial ^2x}>0$ the equilibrium is stable, whereas for $\frac{\partial ^2U}{\partial ^2x}<0$ the equilibrium is unstable. Is there a general solution to this case, or does each have to be considered individually?
What is not clear to me is the case where $\frac{\partial ^2U}{\partial ^2x}=0$. Does this simply mean that the equilibrium is stable given a displacement in one direction and not the other, or is it more complicated - for example if a particle were to oscillate about a stable equilibrium point, its motion would be dampened until it were at rest, but this would not be possible at a point where $\frac{\partial ^2U}{\partial ^2x}=0$; if the particle were to move to the side where $\frac{\partial ^2U}{\partial ^2x}<0$, it would not return to the equilibrium point. Is there a general solution to this case, or does each case have to be considered by inspection?