Force of contact is often called the normal force. It's really the force of electrostatic charges in each body acting on the other, but if we model the field around the ground as a plate made of particles each of some charge, they make an electric field that nets only directly away (i.e. normal, hence "normal" force) to the surface. The reason they push away is these are largely electric charge forces from electrons and like charges repel.
The reason the force is only normal is if we assume the particles are all the same in the body and they they are equally distributed across the surface, then some particle in a body has a field that pushes outward in every direction, but the particles all around it push back in every direction they come from, and none of the particles come from the normal direction, or else that wouldn't be the surface. The bad diagram I made has red lines to show how forces cancel to $0$ and the green lines show how they cancel to make a new total force, that only exists on the surface and points directly away from it, which happens to be straight upward in this picture and for the ground. This is why it doesn't push against the direction the ball came from.
% I wanted the spacing, if you are capable of making better diagrams please do, and also tell me what software you use to diagram, mine are always awful
If we look at a ball, we just say that for a small of region/zoomed in enough, it is locally linear and the same as the ground, which is a plane. So a ball has a field that pushes directly away from the surface, which means it must pass through the center of the ball. And if the ball is a perfect sphere, then contact between the plane and the ball is only at one point and the plane is tangent to the ball. It is a property of tangent to be normal to the radius. So if the plane is tangent, and the force coming off of it is normal, then the force of contact (the normal force) acts along the radius of the ball. This can be seen in the following diagram.
% I wanted the spacing
So when the ball comes into contact, the plane pushes on the ball completely away from the plane. Since the ground is a horizontal plane, it's normal force is completely upward so the ball must interact with the ground going completely upward. Interestingly, the plane feels the opposite, a ball comes along and pushes it down, but it's in the exact opposite direction. This is known as Newtown's Third Law, or the Weak Law of Action and Reaction. It states, "Every action has an equal and opposite reaction." It is weak as in it does not hold up in all cases, you could also research the strong law if you like, it covers some magnetic forces, which do not behave how Newton predicted.
However, all of this ignores friction. Friction is what happens when we realize that those electric fields are not perfect, there's little regions where the forces can't cancel perfectly to only go upward because the surface isn't perfectly flat. When we include friction we get some neat things happening, most interesting is that we can get things to spin around when they strike the ground, really useful on the basketball court if you want to show off. Push the ball off your hand with a lot of spin and you can get it to change spinning direction after it comes into contact with the court, if you do it right you can get the ball to return to you if you push it away spinning toward you, but there's a limit on how hard you push it away from you, the ball will eventually need to spin too quickly to get back to you and will end up sliding. But sliding friction is smaller than static friction and that means the ball won't get be pushed back to you as hard and might not be able to change direction.
In reply to answering what happens for friction, we need to cover more than just a picture. The first thing to point out is that in the old scenario we had a perfect ball, and a perfect ball wouldn't deform at all, which you won't find in the universe, but let's say that we can approximate the atomic structure to not change position. When the surfaces come near contact the electric fields cause the force of contact increases, slowing the ball's fall until it pushes it back up. This would take at least some period of time. And if we go with our frictionless model, then that means the ball must have traveled sideways as it pushed into the surface because nothing stopped it from going sideways, therefore it must have continued going on its way. So it needed to slide on the frictionless surface. This alone is enough to continue on and describe how friction works, but we can add a little better explanation of it if we let the ball and/or plane bend a little bit or deform when they come into contact. The contact between a perfect ball and plane is a point, this is how we define tangency, but if it squishes, then we'll get a small area of a likely irregular shape. And this area can have defects of its own, unlike the perfect ball. So here's a picture of one surface coming into contact with another surface.
Notice how we get multiple little forces pushing in non specific directions? Friction is essentially the factor that describes the overall effect of these little forces. For the most part, the forces on the ball will come directly off the plane, but these small surface defects will also have an effect, the former is the contact force, or normal force, the other is called friction. From this, we can see that both of these are normal forces, but it's easy for me to say that the ball comes into point contact with the plane, but there will be also be a force that wants to make the ball both slow down and rotate. It's more difficult to describe the normal force as a function of the two surfaces, I have to define these two surfaces and how they bend when they come into contact.