Suppose you know that a constant force $F$ is applied for a distance $d$ then you know the work done is equal to $Fd$ - no further information is needed.
But suppose I tell you that a force $F$ is applied for a time $t$, how are you going to calculate the work done? The problem is that the work done depends on the velocity of the object because it is given by:
$$ W = \int_0^t F~v(t')~dt' $$
So you need to know the velocity $v(t)$ as a function of time. So for example for a free mass you need to know the initial velocity at time zero and you need the mass of the object so you can calculate its acceleration and hence how the velocity changes with time. If the object is not free, e.g. if it's sliding on a rough surface, you now also need all the details of the frictional and drag forces.
Of course this calculation can be done, but it turns a very simple calculation into a potentially very complicated one. That doesn't mean we would never consider the motion as a function of time. For example the force could be time dependent or the mass could be time dependent (this happens in rocket flight as fuel mass decreases). But if there's a simple way to do a calculation you should always choose the simple way, and learning how to spot the simplest way is an important skill for physicists.