Do unstable equilibria lead to a violation of Liouville's theorem? Liouville's theorem says that the flow in phase space is like an incompressible fluid. One implication of this is that if two systems start at different points in phase space their phase-space trajectories cannot merge. But for a potential with an unstable equilibrium, I think I've found a counterexample.
Consider the potential below (excuse bad graphic design skills).

A particle starting at rest at point A, $(q,p) = (x_A,0)$ at $t = 0$, would accelerate down the potential towards the left. Because it has the amount of energy indicated by the purple line, it would come to rest at the local maximum B at $t = T$, an unstable equilibrium $(q,p) = (x_B,0)$. However any particle started at rest at the top of the local maximum B at $t = 0$ would also stay that way forever, including up to $t = T$. Thus there appears to be two trajectories that merge together in violation of Liouville's thorem.
 A: The fact that trajectories cannot merge (intersect)  in the space of phased has nothing to do with the Liouville theorem, but it is a  consequence of the fact that the Hamilton equations are of the first order (and the Hamiltonian us Lipschitz continuous). Keeping the first order of the equations and changing their form arbitrarily  would make false the Liouville theorem, but the trajectories would not merge (or intersect) however.
Secondly, your example is misleading since you are looking at the projection of the trajectories onto the 1D configuration space instead of studying them in the 2D space of phases (where the Liouville theorem holds en passant). If you stay in the space of phases, you see that the solutions of the Hamilton equations  you consider do not intersect. One is just a point (with $p=0$ and $x$ coinciding with the tip B of the potential) and the other is, a  trajectory (closed under some conditions) around that state. When the motion passes through B in the variable $x$, there $p\neq 0$ so that the trajectories do not intersect. It apoarently happens only in rhe configuration space.
A: Disclaimer:  Much of the below answer is taken from another answer of mine at https://physics.stackexchange.com/a/177972/59023.
Definitions and Background
Let us define the density of particles in a volume element, $d \mathbf{x} \ d \mathbf{v}$, at a fixed time, $t$, centered at ($\mathbf{x}$, $\mathbf{v}$) as the quantity $f(\mathbf{x}, \mathbf{v}, t)$.  We assume this function is non-negative, contains a finite amount of matter, and it exists in the space of positive times and $\mathbb{R}^{3}$ and $\mathbb{R}{\scriptstyle_{\mathbf{v}}}^{3}$, where $\mathbb{R}{\scriptstyle_{\mathbf{v}}}^{3}$ is the space of all possible velocities.  Then we can see that there are two ways to interpret $f$:  (1) it can be an approximation of the true phase space density of a gas (large scale compared to inter-particle separations); or (2) it can reflect our ignorance of the true positions and velocities of the particles in the system.  The first interpretation is deterministic while the second is probabilistic.  The latter was used implicitly by Boltzmann [Villani, 2002, 2006].
The Vlasov equation is the collisionless form of the Boltzmann equation.  The Vlasov equation can be written as:
$$
  \frac{ \partial f_{s} }{ \partial t } + \mathbf{v} \cdot \nabla f_{s} + \frac{ \mathbf{F} }{ m_{s} } \cdot \nabla{\scriptstyle_{ \mathbf{v} }} f_{s} = 0 \tag{1}
$$
where $f_{s} = f_{s}(\mathbf{x}, \mathbf{v}, t)$ is the particle distribution function of species $s$, $\mathbf{F}$ is an external force, and $\nabla_{\mathbf{v}}$ = $\hat{\mathbf{x}}$ $\partial$/$\partial v_{x}$ + $\hat{\mathbf{y}}$ $\partial$/$\partial v_{y}$ + $\hat{\mathbf{z}}$ $\partial$/$\partial v_{z}$.  We can modify Equation 1 by using the Lorentz equation for $\mathbf{F}$ for electromagnetic fields and introducing an linear perturbation for any variable quantity of the form A $\rightarrow$ $\langle A \rangle + \delta A$.  Here we have assumed that $\langle \ \rangle$ is an ensemble average and $\delta$ represents fluctuations about the mean ($\langle \delta A \rangle$ = 0).  The result is known as the Vlasov-Poisson-Fokker-Planck model, where the left-hand side of Equation 1 is the Vlasov-Poisson (Vlasov-Maxwell if electromagnetic fields are present) part and perturbations of $\langle f \rangle$ and $\delta f$ due to $\delta \mathbf{E}$ and $\delta \mathbf{B}$ create an effective collision term analogous to the Fokker-Planck collision operator (e.g., BGK operator).  Therefore, we let the following be true:
$$
  \begin{align}
    f_{s} & \rightarrow \langle f_{s} \rangle + \delta f_{s}  \tag{2a}  \\
    \mathbf{E} & \rightarrow \langle \mathbf{E} \rangle + \delta \mathbf{E}  \tag{2b}  \\
    \mathbf{B} & \rightarrow \langle \mathbf{B} \rangle + \delta \mathbf{B}  \tag{2c}
  \end{align}
$$
Liouville's Equation
We know that $\langle f \rangle$ (we imply subscript $s$ but have dropped it out of laziness) satisfies Liouville's equation, or more appropriately, $\partial \langle f \rangle$/$\partial t = 0$.  In general, the equation of motion states:
$$
  \frac{ \partial f }{ \partial t } = f \left[ \left( \frac{ \partial }{ \partial \mathbf{q} } \frac{ d\mathbf{q} }{ dt } \right) + \left( \frac{ \partial }{ \partial \mathbf{p} } \frac{ d\mathbf{p} }{ dt } \right) \right] + \left[ \frac{ d\mathbf{q} }{ dt } \cdot \frac{ \partial f }{ \partial \mathbf{q} } + \frac{ d\mathbf{p} }{ dt } \cdot \frac{ \partial f }{ \partial \mathbf{p} } \right] \tag{3}
$$
where we have defined the canonical phase space of ($\mathbf{q}$, $\mathbf{p}$).  If we simplify the terms $dA/dt$ to $\dot{A}$ and let $\boldsymbol{\Gamma} = (\mathbf{q}, \mathbf{p})$, then we find:
$$
  \begin{align}
    \frac{ \partial f }{ \partial t } & = - f \frac{ \partial }{ \partial \boldsymbol{\Gamma} } \cdot \dot{\boldsymbol{\Gamma}} - \dot{\boldsymbol{\Gamma}} \cdot \frac{ \partial f }{ \partial \boldsymbol{\Gamma} } \tag{4a}  \\
    & = - \frac{ \partial }{ \partial \boldsymbol{\Gamma} } \cdot \left( \dot{\boldsymbol{\Gamma}} f \right) \tag{4b}
  \end{align}
$$
where we find that the last form looks like the continuity equation.  If we define the total time derivative as:
$$
  \frac{ d }{ dt } = \frac{ \partial }{ \partial t } + \dot{\boldsymbol{\Gamma}} \cdot \frac{ \partial }{ \partial \boldsymbol{\Gamma} } \tag{5}
$$
then we can show that the time rate of change of the distribution function is given by:
$$
  \begin{align}
    \frac{ d f }{ dt } & = \frac{ \partial f }{ \partial t } + \dot{\boldsymbol{\Gamma}} \cdot \frac{ \partial f }{ \partial \boldsymbol{\Gamma} } \tag{6a}  \\
    & = - \left[ f \frac{ \partial }{ \partial \boldsymbol{\Gamma} } \cdot \dot{\boldsymbol{\Gamma}} + \dot{\boldsymbol{\Gamma}} \cdot \frac{ \partial f }{ \partial \boldsymbol{\Gamma} } \right] + \dot{\boldsymbol{\Gamma}} \cdot \frac{ \partial f }{ \partial \boldsymbol{\Gamma} } \tag{6b}  \\
    & = - f \frac{ \partial }{ \partial \boldsymbol{\Gamma} } \cdot \dot{\boldsymbol{\Gamma}} \tag{6c}  \\
    & \equiv - f \Lambda\left( \boldsymbol{\Gamma} \right) \tag{6d}
  \end{align}
$$
where $\Lambda$($\boldsymbol{\Gamma}$) is called the \textit{phase space compression factor} [Evans and Morriss, 1990].  Note that Equations 6a through 6d are different forms of Liouville's equation, which have been obtained without reference to the equations of motion and they do not require the existence of a Hamiltonian.  We can rewrite Equation 6d in the following form:
$$
  \frac{ d }{ dt } \ln \lvert f \rvert = - \Lambda\left( \boldsymbol{\Gamma} \right) \tag{7}
$$
This equation appears different than the usual version of Liouville's equation because it was not derived from a Hamiltonian.  If the equations of motion can be generated from a Hamiltonian, then $\Lambda \left( \boldsymbol{\Gamma} \right) = 0$, even in the presence of external fields that act to drive the system away from equilibrium.  Note that the existence of a Hamiltonian is a sufficient, but not necessary condition for $\Lambda \left( \boldsymbol{\Gamma} \right) = 0$.  For incompressible phase space, we recover the simple form of Liouville's equation:
$$
  \frac{ d f }{ dt } = 0 \tag{8}
$$
However, Liouville's theorem can be violated by any of the following:


*

*sources or sinks of particles;

*existence of collisional, dissipative, or other forces causing $\nabla{\scriptstyle_{ \mathbf{v} }} \cdot \mathbf{F} \neq 0$;

*boundaries which lead to particle trapping or exclusion, so that only parts of a distribution can be mapped from one point to another;

*spatial inhomogeneities that lead to velocity filtering (e.g., $\mathbf{E} \times \mathbf{B}$-drifts that prevent particles with smaller velocities from reaching the location they would have reached had they not drifted); or

*temporal variability at source or elsewhere which leads to non-simultaneous observation of oppositely-directed trajectories.


Questions and Answers

Liouville's theorem says that the flow in phase space is like an incompressible fluid. One implication of this is that if two systems start at different points in phase space their phase-space trajectories cannot merge.

I am not sure this is entirely true.  If you look at Equation 7 above, you will see the general form has phase space compressibility built into it.

Thus there appears to be two trajectories that merge together in violation of Liouville's thorem.

Contrary to what many may think, the commonly expressed form of Liouville's theorem is a very specific case where none of the examples I listed above are present thus preserving an incompressible phase space.
For instance, look at the Boltzmann equation with the irreversible collision operator on the right-hand side.  That equation presents a situation where $\tfrac{ d f }{ dt } \neq 0$, thus violating the simplified form of Liouville's thorem.
References


*

*Evans, D.J., and G. Morriss Statistical Mechanics of Nonequilibrium Liquids, 1st Edition, Academic Press, London, 1990.

*Penrose, O. "Foundations of statistical mechanics," Rep. Prog. Phys. 42, pp. 1937-2006, doi:10.1088/0034-4885/42/12/002, 1979.

*Villani, C. "Chapter 2: A review of mathematical topics in collisional kinetic theory," pp. 71–74, North-Holland, Washington, D.C., doi:10.1016/S1874-5792(02)80004-0, 2002.

*Villani, C. "Entropy production and convergence to equilibrium for the Boltzmann equation," in XIVTH International Congress on Mathematical Physics, edited by J.-C. Zambrini, pp. 130–144, doi:10.1142/9789812704016_0011, 2006.

A: A few comments on the question you are asking:

*

*The Liouville equation describes not a single particle, but an ensemble of states that evolve according to the laws of mechanics. The Liouville equation says that the entropy of the system stays constant at all times, which is to say that this equation describes isentropic processes and nothing else.  The approach to equilibrium is accompanied by entropy increase, something that the Liouville equation cannot handle.


*The statement that "two systems start at different points in phase space their phase-space trajectories cannot merge" is correct, but it is not a consequence of the Liouville equation. It follows from the fact that the differential equations of motion have a unique solution for given initial condition. Merging would imply that two different initial conditions produce the same state at some later time, which is not possible.


*Your example does not violate the uniqueness of trajectories. To  violate it you need to find two different trajectories in the same potential, with both trajectories starting at $A$ and both ending at $B$.
A: Simply, Liouville's theorem implicates that the two particles can not take the same state at t=T because their state trajectories behave as currents in  incomprisible fluid. 
Physically, when the first particle come to B, it will push the second particle and throw it to the left side. 
