Pressure also has to do with the amount of fluid which is pushing onto the surface you are talking about. In a container, for example, a surface which is at a greater depth has a larger column of fluid above it (i.e. a greater weight pushing down on it.).
The pressure on a surface (parallel to the base/opening of the container, for simplicity) at some depth $h$ is calculated as: $$P=P_o +\rho gh$$ (where $g$ is the gravitational field/acceleration in that region and $P_o$ is the atmospheric pressure.)
If you mess around with this formula a bit you will realize that this quantifies pressure in terms of the weight of the fluid column above the surface you choose.