This is how I think of what work is.I am sure I am wrong somewhere because I shouldn't be coming to the conclusion that I am coming to.It would be helpful if you would point out where this conceptual misunderstanding is.

Work is just change in the energy of an object.The only way an object can gain energy is by movement.Basically if an object's velocity increases then we can say that work has been done on the object.So, work done on an object is directly proportional to the change in its velocity.Also, if the object's mass is high, then for a given change in the objects velocity, the object gains more energy than an object with lower mass, because higher the mass more will be the momentum it can transfer to other objects and therefore higher the mass of the object for a given change in velocity, more is the work done on the object.Therefore work done is directly proportional to the mass of the object.

I don't seem to find any other factor that influences the work done on an object .Hence according to me work should be equal to mass times the change in velocity.