can't we say that from the moment the ball starts to touch the wall, the force of the wall is displacing the ball and thus it is doing work?
Looking at the picture there does not seem to be any displacement of the point of contact between the wall and the ball. The ball is deforming, but the plane of contact does not displace. Because there is no displacement of the wall in the direction of motion there is no work done.
The various parts of the ball have substantial displacement. So there is a lot of work being done by one part of the ball on another. But that is all internal forces within the ball.
Another definition of work is energy transferred from one object to another. Since no energy is transferred, all of the ball’s energy stays within the ball. The energy within the ball changes rapidly from kinetic energy to elastic potential energy and back to kinetic. Thus the ball bounces back elastically. Perhaps a little energy goes into heat and sound, but it appears that almost none goes into the wall.