But the individual parts of the sponge ball are moving from their position.
However if you are holding a pre-compressed ball, and maintaing the compression, then no work is done.
Your argument might have been true if the ball was rigid and compressible. But it's not.