Why is energy calculated with respect to distance and not time? Okay, so looking at the basic definition of energy the force is summed over the distance it is applied. Why exactly is it taken over the distance applied and not the time applied? I understand that the impulse and hence the change in momentum is what we call this summation over time, but it's not exactly clear to me why we chose to do it the way we did. 
Side note - I've seen the examples where there's an object at rest and if you took the force over time you would get infinite energy, but if you took the sum of the force (both holding it up and pushing it down (gravity) you would get 0, and thus an integral over the net force would be 0, right? Thanks 
-edit
Okay, so really the crux of what I'm getting at that I can't seem to find an answer for is: Why did we choose to do it this way? What experiment or thought experiment led us to believe that momentum isn't in fact energy, but a separate quantity? The problem I keep having is that when you want to sum up this quantity we know as force, you have two options, sum it over the time its applied or the distance. I just don't understand the idea behind choosing one over the other.
 A: Because the integral of force over time is impulse, the change in momentum. Momtentum is also conserved, but it is a very different concept from energy, inasmuch as momentum in one direction can cancel momentum in the opposite direction, but energy never cancels out, to my knowledge, it just gets transferred around/changes form. 
The recognition of energy as a conserved quantity that is conserved separately from momentum is an interesting chapter in the history of physics that involved experiments where the penetration depth of a ball falling into clay more closely corresponds with energy than momentum (See the vis viva debate).
A: Your body is acted on by the force of gravity all the time, but unless you jump out of the window your body does not gain any kinetic energy. Summing force over time doesn't give a meaningful quantity, whereas summing force over distance gives us the increase in kinetic energy.
A: When you lift a book to a high shelf, the distance makes the difference. The bigger the distance to the floor, the more energy is stored. 
No matter how much time that passes, this stored energy is the same. It doesn't depend on time, it depends on distance. 
