Does the displacement in the definition of work correspond to the displacement of the object or the point of application of the force? Work is defined as
$$W = \vec{F}\cdot\vec{s}$$
But what what exactly is $\vec{s}$? Is it the displacement of the body on which the force is being applied? Or is it the displacement of the point of application of the force? 
If you look at the derivation for the definition of work from kinetic energy (the way many textbooks do it), it would seem to be the former, but I'm not sure since I can think of many examples that contradict the fact.
Or is it something else entirely? 
 A: Here's a way to argue it isn't the displacement of the (center of mass) of the body.
Take a spring that's at rest. Apply two forces on either end so it compresses. You can do this in such a way that the center of mass of the spring doesn't move. However, the energy of the spring system has changed (the potential energy increased). In order to satisfy the work energy theorem
$$W_\text{net, external} = \Delta E_\text{total},$$
it would seem that the work must be non-zero. If you used the center of mass displacement, then $W=0$, which is inconsistent. The solution is to use the displacement of the point of application of the (two) force(s):
$$W=W_1^\text{app} + W_2^\text{app}.$$
A: First of all, I appreciate your doubt. It is conceptual and involves critical thinking regarding the topic.
In order to understand work, you need to understand the 2 physical quantities namely - Force and Displacement .
Note that using only one single word Displacement in order to define work is not appropriate. (Same is for Force ) . As, Displacement depends on the direction of Force . Now, again, it will not be appropriate to just write Force in order to define Work/Displacement. I will better write it as : As, Displacement depends on the direction of Force applied on the point of application. 

Whenever a body is moving, there are many forces acting upon it. While you are walking, the body is applying force on you, the ground is applying force on you, the air is resisting you from moving etc. So, basically it is much more complicated than we think.
Coming to the main point, displacement! Let us just take a point object in order to understand the situation well. Now, before I go on, I will first define what is point-object. 
The concept of a point object can be understood here : Point Particle
So, as you're clear with the basic concepts, I can go on. Each object is made up of millions of particles...Uncountable! Now, when you push an object, suppose a car, then the car is moving but the force has acted upon on a specific point. That point may be referred to point of application of the force. When the car moves (if the other opposing forces become lesser than the force applied by you) , then it surely displaces itself. Thus, it does some Work. While, you have also done the work to push the object, but we usually calculate the work done by the object experiencing the force. 
The displacement here, will be based on the point particle though many simply refer it to the complete object. It is not wrong either, but just a little mis-conceptional to refer the displacement to a whole object. You must keep this in mind that basically, the force(external) is acting upon the point particle which later on affects the nature of the whole object, that's why the object displaces itself. 
This can be explained in 1-2 para also, but, what I'm trying to explain is critically-thinking a topic. Whenever you try to follow a question or topic, first think about it. Try to compare the conditions with nature, notice the similarities and differences. This will make your concepts much stronger!
A: It is the displacement of the point on which the force is applied. This is it. That is the real definition, period. The books which do it otherwise are wrong. I would like you to read Resnick Halliday Krane , physics Vol. 1, it has a separate chapter devoted to this problem with most of the books available.
As an example to show that this is what the definition should be, you write an infinitesimal work done as, you could not use things like $m\frac{d^2x}{dt^2}dx$, the $d^2x$ and $dx$ would then be for different points and the integration wouldn't proceed the way it does.
Other simplifications arise from constraints, newton's 3rd law and viewing from different reference frame when applied to complex bodies.
I would still advise you to at least once go through that marvellous book. It is the best book on introductory physics that exists.
A: It's the displacement of the point of application of the force.  Elementary textbooks initially model objects as point particles.  They have exactly two properties: position and mass.  No others.  So the displacement of the object is necessarily the same as the displacement of the point of application.
Later, extended or compound objects may (or may not) be introduced.  Now the object has more properties:  a size, an internal structure, maybe others.  And it may not be rigid.  We usually define the position of an extended object as the location of the center of mass.  A force can be applied to some part of the object, and the displacement of that part is not necessarily the same as the displacement of the center of mass.  In those cases, the work done is the force times the displacement of the point of application, not the displacement of the position of the object (center of mass).  Note the Kinetic Energy theorem fails to be applicable in such systems!
