Why does work equal force times distance? My book says:

Energy is the capacity to do work and work is the product of net force and the 1-dimensional distance it made a body travel while constantly affecting it.

This seems quite unmotivated to me.
Why does work equal $F \cdot d$ ?
Where does the distance part come from?
I always thought of time as the one thing we can only measure (not affect) so it justifies why we may measure other things in relation to time. But we have a much greater control over distance (since it's just a term for a physical dimension we can more or less influence as opposed to time).
Edit: That^ doesn't make much sense, but it's been graciously addressed in some of the answers.
Level: High-school Physics.
 A: To add to orbifold's answer, I'll give a quick repeat of Feynman's version of the conservation of energy argument. This is "d'Alembert's principle" or "the principle of virtual work", and it generalizes to define thermodynamic potentials as well, which include entropy quantities inside. 
Suppose you have a bunch of masses on the Earth's surface. Suppose you also have some elevators, and pullies. You are asked to lift some masses and lower other masses, but you are very weak, and you can't lift any of them at all, you can just slide them around (the ground is slippery), put them on elevators, and take them off at different heights.
You can put two equal masses on opposite sides of a pulley-elevator system, and then, so long as you lift a mass up by a height h, and lower an equal mass down by an equal height h, you don't need to do any work (colloquially), you just have to give little nudges to get the thing to stop and start at the appropriate height.
If you want to move an object which is twice as heavy, you can use a force doubling machine, like a lever with one arm twice as long as another. By arranging the heavy mass on the short arm, and the light mass on the long arm, you can move the heavy mass down, and the light mass up twice as much without doing any work.
In both these processes, the total mass-times-height is conserved. If you keep the mass-times-height constant at the beginning and at the end, you can always arrange a pulley system to move objects from the initial arrangement to the final one.
Suppose now that the gravitational field is varying, so that some places, you have a strong "g" and other places a weak "g". This requires balancing the total force on opposite sides of the elevator, not the total mass. So the general condition that you can move things without effort is that if you move an object which feels a force "F" an amount "d" in the direction of the force is acting, you can use this motion plus a pulley system to move another object which feels a force "F'" an amount "d'" against the direction of the force.
This means that for any reversible motion with pullies, levers, and gears
$$\sum_i F_i \cdot d_i = 0 $$
This is the condition under which you don't have to do colloquial work to rearrange the objects. One can take the conserved quantity for these motions to be the sum of the force times the distance for each little motion, and it is additive among different objects, and so long as nothing is moving very fast, if you add up the changes in F dot d for all the objects, it must be zero if you did everything reversibly.
This generalizes to a dynamical situation by adding a quantity of motion which is additively conserved along with F dot d, this quantity is the kinetic energy. You can also go backwards, and start with the kinetic energy idea (which can be motivated by collisions), and re-derive the F dot d thing. These are two complementary points of view that fit together to give a coherent picture of kinetic and potential energy.
if you have a static force field on a particle which has the property that along some closed cycle the sum of the force times the little displacements is not zero, then you can use this cycle to lift weights.
The proof is simple: arrange a pulley system to lift/lower weights at every point along the cycle in such a way that the F dot d of the weights balances the F dot d of the force. Then take the particle around the loop in the direction where F dot d is net positive, while balancing out the force with the weights. At the end of the day, you lifted some weights and brought the particle back where it started.
This means that a non-conservative force can be used to lift a weight. As you traverse the loop, something must be eaten up out of the non-conservative force field, otherwise it is an inexhaustible source of weight-lifting, and violates the first law of thermodynamics. So eventually, all force fields settle down so that the integral of F dot d is zero along every loop. This is the definition of a conservative force.
A: You should regard this passage as a perhaps ill motivated definition of what work is for the purpose of your physics course. Let me make some remarks on energy though, since the passage "Energy is the capacity to do work" is misleading at best. There is a law or principle, called conservation of energy governing all natural phenomena we are aware of. According to this law there is a quantity called energy that does not change during any changes that nature undergoes. It is not tied to anything concrete like pushing boxes around, instead it is an abstract concept. 
There are different forms of energy, among them:
gravitational energy, kinetic energy, radiant energy, nuclear energy, mass energy, chemical energy, heat energy, elastic energy, electrical energy.
Notice that those are just other names though, no one really knows what energy is, there are just different ways of calculating contributions to it. 
For a given physical system the different forms of energies can sometimes be given by concrete formulas. But it is important to realize that conservation of energy is independent of that knowledge. As time goes on, the different forms of energy are converted into each other, conservation of energy means that their sum remains constant.
Now in general energy that is measured relative to the location of something else is called potential energy. Examples are the gravitational potential energy or the electric potential energy. How do you change the potential energy of an object? By moving it around. So how does force come into play? Well it turns out that as a general principle:
$$ \{ \text{Change in potential energy} \} = \textrm{(force)} \times (\text{distance force acts through})$$
The reason for this formula is rather simple, if $V$ denotes the potential energy, you can compare it at two neighbouring points $P$ and $P + \delta P$, then per definition the infitesimal (that means that you neglect terms with $\delta P^2$) change of the potential is:
$$\delta V = V(P + \delta P) - V(P) = F \cdot \delta P$$
Now you just have to sum up those contributions. To give you a simple example: Take the potential energy $V(x) = \frac{1}{2} k x^2$, then
$$V(x + \delta x) - V(x) = \frac{1}{2} k (2 x \delta x + \delta x^2) = kx\delta x$$
So the force is $F = kx$, which you might recognize as Hooke's law.
If you want to read a much better description of this you should read chapter 4 in Volume 1 of Feynman's Lectures on Physics. Maybe you can get a copy of them in your local library.
A: The problem with the definition of the work is that our intuitive idea of work is simply wrong.  Imagine for example, that you hold 10 kg box 1 m above the ground for an hour.  You'd probably feel very tired and you'd think "what a work I've done".  But you could easily put the box on a table, that is 1 m high.  The effect on the box would be just the same.  But did the table made any work?  Of course not.  Since table at rest does not have energy (capacity to do work), it could not possibly make any work.  So you see, there is absolutely no direct correspondence between work and time.
An object therefore must be moved to actually make a mechanical work.  And even in this case you can move object and still make no work.  Imagine that you are carrying 10 kg box at the height of 1 m from one side of the room to the other side.  It actually requires making some effort to do that.  But you could put the box on the cart 1 m high and just gently and slowly push it over the room.  But did the cart made any work?  The answer is again negative.  Since cart practically at rest does not have energy (capacity to do work), it could not possibly make any work.
Only the product of force and displacement in the direction of the force is a meaningful work.
A: Fortunately, we've equipped ourselves with rulers in addition to clocks, so it's quite possible to measure time and distance, and quite useful to do both. Let's push something (with a force $F$ ) so that its velocity changes by $\Delta v$ and figure out how much its energy changes. Say our object is initially moving at speed $v_0$ and has energy 
$$E_0 = \frac{1}{2} m v_0^2$$
After the push, it is moving at $v = v_0 + \Delta v$ so its energy will be:
$$E = \frac{1}{2} m (v_0 + \Delta v)^2$$
$$E = \frac{1}{2}m(v_0^2 + 2v_0\Delta v + \Delta v^2)$$
$$E = \frac{1}{2}mv_0^2 + m v_0\Delta v + \frac{1}{2}m\Delta v^2$$
Now with $\Delta v$ small enough so that we can ignore the last term with $\Delta v ^2$:
$$E = E_0 + mv_0\Delta v$$ or in terms of the change in energy:
$$\Delta E = m v \Delta v$$
There are a few different ways to proceed from here, one is to multiply by $\Delta t / \Delta t = 1$ and rewrite as:
$$ \Delta E = m \left( \frac{\Delta v}{\Delta t} \right) (\Delta t) \cdot v$$
Now recognizing from Newton's 2nd law that 
$$\frac{m \Delta v}{\Delta t} = \frac{\Delta (mv)}{\Delta t} = \frac{\Delta p}{\Delta t} = F$$
We have that
$$\Delta E = F \Delta t \cdot v$$ but 
$\Delta t \cdot v = \Delta t \left( \frac{\Delta x}{\Delta t} \right) = \Delta x$ is just how far we pushed it, so
$$ \Delta E = F \cdot \Delta x$$
Thinking about the problem as time progresses, it is natural to also ask "How fast is the energy changing?" The answer is that we are supplying power to the system at the rate:
$$ P = \frac{\Delta E}{\Delta t} = F v$$
This may be closer to how you are thinking of it, but as you can see above, if you want to relate the applied force to the change in energy, multiplying the force by the distance over which it is applied gives the correct result. 
A: It's by moving something over a distance that you put energy into it. You can push a wall with all your strength for hours, but if it doesn't move then it hasn't gained any energy. (Instead your energy will have gone into heating up your body and heating the environment around you)
On the other hand if you carry a weight upstairs, then you are applying a force to it and moving it, and the higher you carry it the more energy the weight has and the more work you could make it do by dropping it.
