Why is normal force perpendicular? I realize "normal" is just another word for perpendicular, but semantics aside, what causes normal force to be always perpendicular to the plane?
For example, on an inclined plane, why is it that the plane pushes the box outwards rather than directly against gravity? It seems to me that in order for there to be normal force, there must be a force trying to push the box directly into the plane, but gravity is going directly downwards, not into the plane.

 A: Newton's third law: when two bodies interact, they apply equal but opposite forces on one another.
Whenever you go to apply this, you have to be really careful that you're consistent about which two bodies you're talking about.
The weight is the force that the Earth as a whole applies to the box. It has nothing to do with the ramp. By Newton's 3rd law, the box also pulls the Earth up a tiny bit, but we're ignoring that in this problem because we're not considering how the Earth moves, and besides a little box barely matters to the Earth's motion.
So the interaction we're concerned about in this problem is between the box and the ramp. The box pushes down and to the right on the ramp, and the ramp pushes up and to the left by an equal amount. That satisfies Newton's 3rd law.
We choose to  decompose the weight vector into components along the ramp and perpendicular to the ramp in order to make it easier to solve this problem. The box is not accelerating through the ramp, although it may slide along the ramp. That tells us that the force components directly into and out of the ramp must be balanced. I.e. they must add to 0. Only the components along the ramp may or may not be 0, depending on whether or not the box is sliding.
A: "Normal" is a mathematical synonym for "perpendicular"
addendum after comment
Yes, I read the question too fast.  The normal force is caused by very small compressions of the intermolecular bonds connecting the first layer of molecules to the second.  If those bonds are perpendicular to the surface, it's clear that the force will be perpendicular.  Generally the bonds don't all point normal to the surface.  However, on average, the horizontal components cancel, leaving only the perpendicular force.
A: Let's talk about what the normal force is.
First of all, it is a feature of solids, which is to say materials that resist penetration by other materials. When two solids are in contact they resist interpenetration; they resist occupying the same space.
Now, if something (be it gravity, your own hands, or simple motion) brings two solids toward one another and they are prevented from moving into the volume occupied by the other there must be a force involved. Where it comes from on the molecular level is complicated, but on the human level it is simply an expression of the resistance of solids to occupy the same space.
We define "the normal force" as that force which resists an attempt to cause two solids to occupy the same space. As such it points perpendicularly to the surface of contact; because motion along the surface of contact is not interpenetrating. 

Of course there is a force related to motion along the surface of contact, too, but it goes by a different name---friction---and follow different rules.
A: If your ramp has friction holding the block in place, then you could talk about the total force the ramp applies to the block, and this total force would point directly opposite the force of gravity, and it would have the same magnitude, giving a net force of 0. It might be more useful to consider the case where there is no friction whatsoever. In that case, there can be no force resisting the sliding of the box down the ramp. However, if your normal force somehow pointed in a direction other than perpendicular to the ramp's surface, then there would be a component of the normal force reducing the block's acceleration down the ramp (assuming you make your normal force point up), but that's friction's job, right? If there's no friction, there is no force parallel to the ramp besides a component of the force of gravity pulling the block downward.
A: I think it is obvious in the diagram that you can move the box along the plane without "disturbing" it (lift it a fraction of a mm so that they are not in contact anymore and move your box freely along the surface and you will not make any effect on the plane), this is the direction in which the friction would act, the sliding motion.
If we now turn to the other possibility, perpendicular motion, you can see that we do affect the plane. Namely, we are applying a force that compresses the material at that point. Since the atoms inside the material are "happy" at their current distance (solids keep their shape when put in a container because their atoms are "happily linked together") trying to push them together will cause a force, which will oppose the movement of the box which is going in the normal direction. It follows that the force should go in this same direction.
A: For example, on an inclined plane, why is it that the plane pushes the box outwards rather than directly against gravity? 
The plane does push against the box directly upward against gravity, See the free body diagram below (force components not shown to scale).
In the diagram, I am showing the reaction of the plane directly upwards against gravity. Any such force can be resolved into perpendicular and parallel components to the plane. In this case the normal component is that perpendicular to the plane. But the parallel component (what I show as ??) can only be the friction force, since that is the only force the plane can exert parallel to its surface. That is, for a frictionless surface the parallel component of the force would have to be zero.. 
Hope this helps.

