I'm sure everyone has had that concern when we encountered the definition for the first time, in school.
There is a valid reason why this definition is still persisted with, despite the deficiency that you hit on. The most popular (and simple) forces in physics (also the ones with which we begin learning physics) are conservative forces, implying that the (mechanical) 'work' done depends only on the end state, and not on the path followed in getting there. (Imagine a magic force where you spend the same energy in going from your table to the kitchen via the shortest path, or going to Mars first and then going to the kitchen!) In this sort of a situation, it makes sense to be concerned with the displacement (and not the distance covered) under the influence of the force. That's encapsulated in the defining relation $W = \int \vec{F}\cdot\vec{ds}$.
If you account for the fact that both (Newtonian) Gravitational force relation, as well as electrostatic forces both fall into this category of conservative forces, you can imagine that these definitions are sufficient to provide a description of a huge range of known phenomena. However, most forces that you encounter in everyday life aren't eligible for such a simplified description, since they are vastly more complicated. Especially when you interface with biological systems. So, while the physical definition of work looks paradoxical here, it actually isn't, if you adopt this perspective:
(Let me construct a new term to keep things distinct from the physical work.) The ''un-physical'' work $W_{\rm unp}$ would still be the negative of the energy you spent biologically, minus the heat energy you are contributing to the universe. i.e. we have $\Delta E = H + W_{\rm unp}$. This work could refer to, e.g. the gravitational potential energy gained by lifting something up, in which case, it really is the convenient 'work' of ours. However, even when it isn't, you can easily see that this is a ''one-way definition'', since you always keep spending energy to do work, unlike conservative systems (e.g. when you throw a ball up, it gains energy on the way up and loses that much on the way down. That doesn't happen here). If you walk around your building $n$ times, you are accumulating $\Delta E$, because both $H$ and $W_{\rm unp}$ increase. (For imagining the second one, suppose I fold your circular path into a straight path of length = total distance covered. Then you are doing work even as per the above definition. Notice that the sign of this $W_{\rm unp}$ won't reverse like the gravitational analog. So, it upstream the physical force, is as good as downstream this force). Clearly, apart from verifying your intuitions, there is nothing else physically useful that can be drawn from this definition.