Do all conservative forces act in the direction of decreasing potential energy? Let us take two unlike point charges $q_1$(positive charge) and $q_2$(negative charge). When we decrease the distance between them, the potential energy between them also decreases according to the formula $U=k(q_1q_2)/r$.
Here the conservative force is the electrostatic force which is responsible for the decrease in potential energy.
Can we say that all the conservative forces act in the direction of decreasing potential energy? 
 A: For a given potential $V$, the corresponding conservative field can be found by taking the negative of the gradient of the potential. Mathematically,
$$\mathbf E=-\nabla V\tag{1}$$
where $\mathbf E$ is the conservative vector field corresponding to the potential $V$ (to know why, see this question). In one dimensional cases, equation $(1)$ simplifies to
$$\mathbf E=-\frac{\mathrm d V}{\mathrm d r}\mathbf{\hat r}\tag{2}$$
It's the property of the gradient operator that the unit vector of a scalar function's gradient, points in the direction where the function increases the most. Thus the negative of the gradient will give us the direction in which the function decreases the most. Applying this to equation $(1)$, we can conclude that any general conservative vector field points in the direction where the potential decreases the most.
The physical reason why this happens is because an isolated system tends to minimize its potential energy and thus the particles in the system tend to move towards a lower potential energy. We explain this movement by using the notion of force on the particles due to the (conservative) field acting on them.

In vector calculus, we define $\nabla$ as a vector which is equal to
$$\nabla=\frac{\partial}{\partial x}\mathbf{\hat i}+\frac{\partial}{\partial y}\mathbf{\hat j}+\frac{\partial}{\partial z}\mathbf{\hat k}$$
Thus applying this to a scalar potential $V$, we get
$$\nabla V=\frac{\partial V}{\partial x}\mathbf{\hat i}+\frac{\partial V}{\partial y}\mathbf{\hat j}+\frac{\partial V}{\partial z}\mathbf{\hat k}=-\mathbf E\tag{3}$$
In case of a field which depends only on one of the coordinates (let's say $x$, without any loss of generality), we encounter the one-dimensional case (equation $(2)$) mentioned above and using equation $(3)$, we again get the same equation (because partial derivatives with respect to $y$ and $z$ in this case are $0$)
$$\mathbf E=\frac{\partial V}{\partial x} \mathbf{\hat i}$$
Vector calculus is quite a vast topic and this answer is too small to contain it all. You would like to check out Introduction to Electrodynamics by David Griffiths if you want a more detailed explanation of the concepts of vector calculus.
