You are right. Microscopically, work and heat are just about the same. Both involve molecular collisions transfer energy from one object to the other.
Work involves a kind of "coherent" transfer in a manner of speaking, in which the collisions are predominantly, and to an extreme degree, in one direction. Also, typically the force is applied to one location on the object. And importantly, the boundary of the system is displaced. (E.g. translation or deformation)
On the other hand, transfer of energy by heat is "incoherent", many directions, and typically in all directions. And importantly, the boundary of the system is not displaced.
Finally, everyday phenomena fall into one or the other category, and they differ in their macroscopic behavior. Loosely speaking, when heat is transfered the temperature rises. When work is done, the boundary of the system changes. Of course adding heat generally also results in the boundary moving [say, expanding], and work generally results in a temperature change [Joule's experiment]. I'm trying to motivate the macroscopic separation between work and heat without a lengthy discussion.
To the engineers who first worked all of this out, the very existence of atoms was unknown. To them, the separation between heat and work was very clear. They had little reason to view them as manifestations of the same microscopic process. They thought that heat was a physical fluid. In any event, their remarkable achievements have stood the test of time.