But if I imagine applying the same amount of force to a light ball that already rolled in quickly over this distance, and compare it with accelerating a heavy ball with the same force over the same distance - I can't imagine that it's the same amount of work.
I have emboldened the word imagine in the sentence above because I think that you are surmising as to what might happen if you actually tried to perform the "experiment".
The work that your body is doing is not the same as the work which is done on the ball(s). You would expend more energy applying a force of $1\,\rm N$ for one minute than for one second. Think of holding up a weight at a "constant" height above the ground where it would appear that you need to do no work but if fact your body must be doing work as you do get fatigued as time progresses. This is probably the most important factor which distorts your imagination of what happens.
So yeah, basically: Why is the Energy transferred by a force dependent on length over which it was applied, and not the time?
Time does come into it as the time the force is applied depends on the speed of the ball.
There is a practical problem of applying a force of $1\,\rm N$ over a distance of $1 \,\rm m$ when the ball is moving. For the large ball starting from rest you can imagine that this can be done relatively easily because the time over which the force need to be applied will be several seconds but what about a ball which is initially moving at speed of $10 \, \rm m \, s^{-1}$? Now the force would have to be applied in under a $\frac {1}{10}^{\rm th}$ of a second. (Almost) impossible to do (a push vs a kick) and hence difficult to imagine doing it under controlled conditions?
Now what might you actually observe?
With the large ball, mass $1\,\rm kg$, which lets suppose started from rest you would easily observe an increase in its speed.
work done = change in kinetic energy $\Rightarrow 1 \times 1 = \frac 12 \times 1 \times \left (v^2_{\rm final} - 0^2 \right) \Rightarrow v_{\rm final} \sim 1.4 \,\rm m\, s^{-1}$
What about the lighter ball, mass $0.1\,\rm kg$, which is moving at $10 \, \rm m \, s^{-1}$ before the force is applied?
work done = change in kinetic energy $\Rightarrow 1 \times 1 = \frac 12 \times 0.1 \times \left (v^2_{\rm final} - 10^2 \right) \Rightarrow v_{\rm final} \sim 11 \,\rm m\, s^{-1}$
Is this an easily observable change in the speed of the ball?
Granted that if the mass of the ball was even small, say $0.01 \,\rm kg$ then the final speed would be nearly double, $17 \,\rm m\, s^{-1}$, and you would probably notice the difference in speed.