There are three parts to the phenomenon, two real and one illusory.
While you are lowering the bat, its relative velocity to the approaching ball increases that little bit. The ball bounces off it that bit harder, gaining twice that extra velocity relative to the floor. Repeat for several bounces and the difference might become noticeable. This is one real part.
The other arises because the ball slows as it rises and accelerates again as it falls. Lowering the bat cuts out the bit where it slows down, so even though the local speed at any given point may not increase, the average speed does increase.
The illusion is to do with the scale and period of the bouncing. As you lower the bat, the period of each bounce shortens, increasing the frequency of the bouncing. This combines with the shrinking scale to create an illusion of going faster. (Credit to user Accumulation for pointing this one out in another answer).
A similar illusion takes place when you watch a scurrying insect. Compare say a horse, a cat and an insect walking along. The big horse seems slow and lazy, the tiny insect in a mad hurry, the cat somewhere in between. But in reality the horse is going the fastest and the insect the slowest.