What's the cause of sideways pressure (and why pressure is scalar)? Since I think the two questions are related that's why I decided to put them together. Why pressure is a scalar ? It's the ratio of two vectors (force and directed area) right ? 
Also, suppose you have a cylinder filled up with water, and at a point $P$ at distance $h$ from the surface, I find it intuive to understand that there would be a horizontal pressure because of the weight of the water in a column with height $h$ erected from $P$, but I don't understand why there should be a sideways pressure, with the same value as the downward pressure at $P$ too ? What's the cause for it ?
Why the pressure is same in all directions ? 
(Please don't use stress tensors/linear algebra to answer my questions since I don't understand them)
 A: It's because water molecules are slippery. If they were not (e.g. compare with atoms in a solid) then the pressure would not be same in all directions.
When some higher up molecule gets pushed down onto one below, it is very unlikely for them to happen to be exactly at a point where the top one could balance on the one below. Instead it slides off to the side. So this will continue until the other molecules off to the side push back against the ones coming down and slipping off. The whole thing rapidly results in the forces getting to act in all directions and balancing one another: the water will only stop moving when the forces do balance. If they do not balance then the water will slip again and flow until the forces get balanced again. And that means balanced to the side as well as up and down.
The reason for the exact balance in an ideal fluid is that there is nothing to prevent this slipping of one molecule over another. They have no sheer force, no stickiness. In a real fluid there is a bit of stickiness, called viscosity, and then the horizontal pressure does not always have to be equal to the vertical pressure at any given point.
A: We can possibly explain it through symmetry. 
Let's consider a cylinder partially filled with water and a piston that is used to compress the water.
In the absence of gravity all water molecules in the cylinder should be evenly distributed and  there is not reason why the pressure - a result of their interactions - would be higher in one direction over other directions. 
If we apply more force to the piston, the pressure in the cylinder will increase, but, still, the molecules will be evenly distributed (they would not now if they got squeezed by the piston or by the walls) and, if so, the pressure will still be the same in all directions. 
If now we take an open cylinder with the gravity present, we can say that a tiny slice of water at the bottom of the cylinder is squeezed by the weight of the water above it and the distribution of molecules in that tiny cylinder should be roughly uniform - the same as if it was squeezed by a piston in the previous example. A tiny difference between the pressure at the top and at the bottom of the tiny cylinder will be too small in comparison with the pressure due to the weight of the water above it and, therefore, could be neglected.
Similar logic could be applied for a tiny cylinder in the middle or at the top: the pressure in any of them would be the same in all directions, being highest for the tiny cylinder at the bottom and lowest for the tiny cylinder at the top. 
A: The idea that the pressure is the force for a given directed area suggests at first that pressure $P=F/A$, a ratio of two vectors, but this is not defined. Instead, say that $F=PA$, where $P$ can be defined, but is not uniquely determined by the two vectors as given. 
Within the mode of your question - there are really two different pressure concepts. 
One of them is exactly the (sorry for mentioning) stress tensor you said you did not understand, but actually you do - enough to see what is going on. The stress tensor is just the relation that gives the vector force on a vector directed area. Strictly, the pressure tensor is the negative of the stress tensor as pressure is the negative of tension (think of pulling a balloon out rather than pushing it in).
The other concept is the scalar pressure, which is the average normal force. The normal force is a scalar (just a number) - the length of the normal force on a unit area as a vector. If this scalar is taken in three orthogonal directions and averaged, this is what is called the pressure. It is also just a number. And since it is the average in all directions - it does not depend on direction.
If, for example, you get a block of rubber and push it down on the top but pull it out from the sides, then the force per unit area will be different for a horizontal surface inside than for a vertical surface. But the pressure will be the average. So, it is possible to have zero pressure even though the forces are not zero - if a small chunk of the material is being pulled and pushed so that the average is zero.
But, while you can do this to a fluid, such as water, the result would be that the water would flow out the side with the lower pressure. So, this could only occur for a short time. Actually, in this case, the inertia of the water starts to be important - that that introduces the idea of dynamic pressure, due to resistance of the fluid to motion. This introduces a number of complications including that the pressure can depend on the speed of the observer. So, there are quite a few conundrums in the pressure concept.
