Hydrostatic pressure? What what I understand, hydrostatic pressure is the "weight" of the water pushing against objects. But if this is true, why is hydrostatic pressure perpendicular to the surface it acts on instead of always going down?
For example, if you placed a book on a desk, the book's weight would push against the desk, but gravity is pulling it "down". But if you put another book beside this book the first book wouldn't apply any force on it. The weight exerted from the book always goes down.
Similarly, why doesn't hydrostatic pressure always go down? I understand that if you put a plate on the seabed, the weight of the water would push down on it, but if you just had a vertical plate standing on the bed, why would force push it from the sides?
To sum up, why does hydrostatic pressure act perpendicular to the object instead of always down?
 A: 
But if this is true, why is hydrostatic pressure perpendicular to the surface it acts on instead of always going down?

Because of the properties of a fluid.  A fluid will not tolerate directionality in force.  If you have a jug of liquid and poke a hole at a surface below the water level, the fluid is ready to flow out in any direction, left, up, down, wherever.  It will have the same push in any direction.
You confusion might be cleared up by trigonometry.  If a surface on the bottom of a tank is slanted, then the force pushing down is the pressure times $\cos{\theta}$, if $\theta$ is the slope.  Mentally deconstruct the force vector into two - a downward component and a sideways.  The downward component fits the model that you want pressure to fit.  It is equal to the weight for fluid above it.  The sideways force is also there, but it doesn't really change things.  Since it's sideways, it won't affect the weight of a jug of water you're holding.  All of the sideways force components will cancel each other out so that the weight felt by the bottom surface is exactly equal to the weight of the fluid in the container.
A: Hydrostatic pressure concers pressure that happens on perfect fluids in equilibrium. A perfect fluid is slippery, devoid of viscosity: when it is in equilibrium, it cannot exert nor resist shear (tangential force). Therefore, on the walls of a vessel sustaining a perfect fluid at rest, the normal component is solely responsible for resisting the weight of the liquid.
