What is the state of the equilibrium for a second derivative equal to zero? Considering a potential energy of $U$, and a displacement of $x$, the force is given by
$F=-\frac{\partial U}{\partial x}$.
Since equilibrium is defined as the point at which $F=0$, we can express this as $\frac{\partial U}{\partial x}=0$. This is clear to see on the following graph;

It is also clear that some equilibria are stable and some are not; given a small displacement at $x_2$ the system will return to equilibrium, whereas this would not happen at $x_3$. Hence, we can say that for $\frac{\partial ^2U}{\partial ^2x}>0$ the equilibrium is stable, whereas for $\frac{\partial ^2U}{\partial ^2x}<0$ the equilibrium is unstable. Is there a general solution to this case, or does each have to be considered individually?
What is not clear to me is the case where $\frac{\partial ^2U}{\partial ^2x}=0$. Does this simply mean that the equilibrium is stable given a displacement in one direction and not the other, or is it more complicated - for example if a particle were to oscillate about a stable equilibrium point, its motion would be dampened until it were at rest, but this would not be possible at a point where $\frac{\partial ^2U}{\partial ^2x}=0$; if the particle were to move to the side where $\frac{\partial ^2U}{\partial ^2x}<0$, it would not return to the equilibrium point. Is there a general solution to this case, or does each case have to be considered by inspection?
 A: First of all you have an imprecise idea about stability: also small velocities matter not only small displacements from the equilibrium. An equilibrium $x_0$ is stable if the motion is confined around $x_0$ and its velocity is confined around the vanishing velocity for every positive time, for every initial condition close to $x_0$ and every initial velocity close to $0$ at time $t=0$. 
In other words, according to the general theory of stability (e.g., see Arnold's or Fasano-Marmi's textbooks) the equilibrium $x_0$ is stable (in the future) if
fixing a neighborhood $U$ of $(x_0,0)$, there exists a second neighborhood $V\subset U$ of $(x_0,0)$ such that, every pair of  initial conditions $x(0)=y_0$ and $\dot{x}(0) = \dot{y}_0$ with $(y_0, \dot{y}_0) \in V$ gives rise to a motion $x=x(t)$ such that $(x(t), \dot{x}(t)) \in U$ for every $t\in (0, +\infty)$.
A theorem (as above I restrict myself to the one-dimensional case) proves that if all forces are conservative then
(a)  a configuration $x_0$ is an equilibrium if and only if $\frac{dU}{dx}|_{x_0}=0$,
(b)  an equilibrium $x_0$ is stable if $U$ has a strict minimum at $x_0$ (i.e. $U(x)>U(x_0)$ for $x\neq x_0$ in a neighborhood of $x_0$).
(c)  an equilibrium $x_0$ is unstable if $\frac{d^2U}{dx^2}|_{x_0}>0$.
The condition in (b) is satisfied if $\frac{d^2U}{dx^2}|_{x_0} <0$, but this is just a sufficient condition (think of $U(x)=x^4$ with $x_0=0$, it is evidently stable and satisfies (b), but $\frac{d^2U}{dx^2}|_{x_0}=0$).
It remains open the case $\frac{d^2U}{dx^2}|_{x_0}=0$. It has to be studied case-by-case. However certain cases are easy. In particular consider any point $x_0 > x_5$ in your picture. It is clear that the condition in (a) is true, so that $x_0$ is an equilibrium and also $\frac{d^2U}{dx^2}|_{x_0}=0$. 
However, perhaps contrarily to the naive idea, $x_0>x_5$  is unstable. Indeed if you start with an initial condition $x(0) = y_0$ arbitrarily close to $x_0$ and a speed $\dot{x}(0) = \dot{y}_0 > 0$ arbitrarily close to $0$, the arising motion is $x(t) = \dot{y}_0 t + y_0$ and, waiting a sufficiently large time $t>0$, $x(t)$ exits from every neighborhood of $x_0$ initially fixed.
(Statement (b) is nowadays an elementary  subcase of a famous theorem due to Lyapunov but a proof was already known by Lagrange and Dirichlet. As a matter of fact, the total energy $E(x, \dot{x})$ is a Lyapunov function for the system for the critical point $(x_0,0)$ when $U$ has a strict minimum at $x_0$.)
A: Taylor expand the force $F(x) = -U'(x)$ about $x = x_0$:
$$F(x_0 + \Delta x) = -U'(x_0) - U''(x_0)\Delta x - \frac{1}{2}U'''(x_0)(\Delta x)^2 + \cdots$$
Stipulate that $F(x_0) = 0$ and then
$$F(x_0 + \Delta x) = -U''(x_0)\Delta x - \frac{1}{2}U'''(x_0)(\Delta x)^2 + \cdots$$
In the case that $U''(x_0) \gt 0$, then for $\Delta x$ small, the force is approximately a linear restoring force.
However, in the case that $U''(x_0) = 0$  (and at least one higher order derivative is non-zero), then for $\Delta x$ small, the force is non-linear and not necessarily a restoring force.
For example, if $U'''(x_0) \ne 0$, then the sign of the force does not change as $\Delta x$ goes through zero; the force is opposite the displacement in one direction and with the displacement in the other direction (which will drive the particle away from $x=x_0$).
For the force to be restoring in the case that $U''(x_0) = 0$ requires that the lowest order non-zero derivative (higher than the 2nd) be even order and positive
A: Consider the following potentials:
\begin{align} U(x) &= x^4 \\ U(x) &= x^6 - x^4 \\ U(x) &= x^4 + x^3  \end{align}
All three of these potentials have an equilibrium point at $x = 0$. All three of these potentials are such that the second derivative of $U(x)$ at this equilibrium point is zero. However, you should convince yourself (perhaps by plotting these potentials) that in the first  case the equilibrium is stable, in the second case it is unstable, and in the third case the equilibrium is, as you put it, "stable in one direction but unstable in the other".
The moral is: knowing only that the second derivative is zero tells us nothing about stability. We need to look at higher derivatives if we want to know more.
