Conservative force as a potencial energy gradient A conservative force $\vec{F}$ is apparently defined as the gradient of a potential energy $U$:
$$\vec{F} = -\nabla\ U$$
I am curious if this definition was originally used to describe a conservative force in relation to potential energy or the other way around.
Where does this relation come from, and how is it that the gradient of potential energy is a conservative force? Also, how can the expressions for gravitational potential energy and elastic potential energy, for example, be derived from this?
 A: We can start at the relationship: $W=-\Delta U$, which is work done by a conservative force.
The math
A (conservative) force $F$ will do this work on an object when doing a displacement $\Delta x$, and $W=F \Delta x$. In the general case, the force might be different at different points as the object is moved (the force of gravity is not constant along larger vertical distances, e.g.), so the force can approximately be written as a function of the displacement (The same goes for potential energy that can be written as a function $U(x)$):
$$W \approx F(x) \Delta x$$
For this to be exactly true (and replace $\approx$ with $=$), we need $F(x)$ to be constant over at least the displacement $\Delta x$. So by considering shorter and shorter intervals of displacement, $\Delta x \rightarrow0$, we can use:
$$W = F(x) \Delta x$$
Combining these:
$$W=-\Delta U \text{ and } W = F(x) \Delta x$$
$$F(x) \Delta x=-\Delta U$$
$$F(x) =-\frac{\Delta U}{\Delta x}$$
And, as mentioned, in the limit $\Delta x \rightarrow0$:
$$F(x) =-\frac{d U(x)}{dx}$$
Here is your result.
Understanding it
This e.g. describes potential wells.
If you put a glass ball in a bowl, it will find the place of lowest potential (the bottom of the bowl) and stay there. If it rolls to either side, it will start an uphill motion increasing $U$, resulting in an opposing force $F(x) =-\frac{d U(x)}{dx}<0$. By moving to either side the ball will be pushed back.
On the bottom, if the ball rolls just an infinitisemal distance to either side the change in potential energy will not be as great, because it is in the bottom of a potential well.
Another example is a stone rolling off a cliff. While it rolls down the cliff-side there might be several flat plateaus. All of these flat points are spots of no change in potential energy, $\frac{d U(x)}{dx}=0$. And if you put the stone gently (without giving it a sideways motion) on such a plateau, it will stay there, since no force tends to pull in it, which is seen from $F(x) =-\frac{d U(x)}{dx}=0$.
A third example is the spring. In the unstretched position nothing happens. If it is moves slightly to either side it will get an increase in potential energy, and the force $F(x) =-\frac{d U(x)}{dx}<0$ will oppose the motion and return it to the unstretched condition.
A: Here is one take on how to understand the relation between force and potential energy, which I think is the closest modern version of how it would have been seen originally. Let's take as the conditions for a force to be conservative 
$$\nabla \times \mathbf{F} = 0$$
and $\mathbf{F}$ is a function of position only (this leaves out the magnetic force on a charge, which requires more sophisticated tools). Stoke's theorem shows that the integral of this force around any closed path is 0, which means that the integral of this force between points $\mathbf{r}_1$ and $\mathbf{r}_2$ is independent of the path taken, or
$$ \int_{\mathbf{r}_1}^{\mathbf{r}_2} \mathbf{F} \cdot \mathrm{d}\mathbf{s} = f(\mathbf{r}_2)-f(\mathbf{r}_1)$$
with $f$ some scalar function. We can take the limit of both sides as $\mathbf{r}_2$ and $\mathbf{r}_1$ become very close to find
$$\mathbf{F}(\mathbf{r}_1) \cdot \Delta\mathbf{r} = f(\mathbf{r_1}+\Delta\mathbf{r})-f(\mathbf{r_1}) = \mathbf{\nabla}f(\mathbf{r}_1) \cdot \Delta\mathbf{r}  $$
We can then pick some point $\mathbf{r}_0$ (often infinity) and define a quantity
$$ U(\mathbf{r}) = f(\mathbf{r})-f(\mathbf{r}_0)$$
whose gradient at a point will be the force at that point. This quantity $U$ turns out to be very useful, because
$$  \frac{1}{2}m (v_2^2-v_1^2) = U_2-U_1$$
This is not too hard to show for a constant force. You can then extend the argument to an arbitrary force by carefully setting up the integral.
So, on this view, it seems that potential energy should be understood as a convenient mathematical trick which can get you speeds without having to go through integration. As with many quantities in physics, it has since taken on a life of its own (like, say, the electric field). Lagrange and Hamilton recast mechanics in terms of energies, and given that all fundamental forces are conservative it's just as plausible to view forces and Newton's laws as convenient mathematical tricks to get dynamics out of complicated potentials. 
How should you view it? There is no physical difference between conservative forces and potential energy. If you have one, you have the other. They are different only in mathematical approach. You should see them as different words for the same thing, and use whichever form is most convenient for the problem at hand.
