To simplify the problem, we could replace a ball by a hockey puck on ice and ask why it bounces back, when it hits a wall, but just slows down in the absence of serious obstacles.
One possible way to think about it is that a puck, as it slides across the ice, undergoes gazillions of micro-collisions with gazillions of micro-obstacles. Each micro-collision takes away a tiny fraction of the puck's kinetic energy as the puck either a) pushes a micro-obstacle out of the way or b) jumps over it.
This outcome of individual collisions is possible because the obstacles are a) easily moved or b) small in comparison to the radius of the puck's edge, which results in an oblique contact and allows the puck to proceed forward with a minor hick-up.
Conversely, if an obstacle was straight and tall enough (in comparison with the radius of the puck's edge) and, at the same time, strong enough to withstand the push, the puck would not have any way to proceed forward and all its kinetic energy would be lost at once, some of it to the heat and some of it converted to the elastic energy stored between the puck and the obstacle, which, in turn, would quickly be converted back to the kinetic energy of the reflected puck.
So, we can say that, although the force of friction is directed against the motion of the puck and the combined work performed during multiple micro-collisions, associated with friction, is equal to the initial kinetic energy of the puck, a specific nature of individual collisions, as described above, makes it impossible to stop the puck at once and send it back.