but work done depends on displacement only
That's not correct, as discussed below.
In physics, distance is a scalar quantity that essentially refers to "how much ground an object has covered" during its motion, whereas displacement is a vector quantity that refers to "how far out of place an object is" , i.e., the object's overall change in position. The general definition of work is
$$W=\int_1^2\vec F\cdot d\vec s$$
We see that work depends on the dot product of the force times the differential displacement of the object, not simply the product of the force and the displacement of the object. The dot product is a scalar quantity equal to the distance covered in the direction of the force between the initial and final position of the object. The work done by a non conservative force depends on that distance.
Case in point is work done against kinetic friction between two points. The direction of the kinetic friction force is always opposite to the direction of the motion of the object. When the force applied to a sliding object changes direction, the opposing friction force also changes direction so that the direction of the force is always in line with (and opposite to) the direction the displacement. Since the angle between the applied force and displacement is always zero, the dot product is unity (positive for the work done by the agent pushing the object and negative for the work done by friction) and thus the magnitude of the total work done is the average force times the total distance covered.
The displacement is a vector that points from the initial position to the final position of the object. When the work done depends only on the initial and final positions, and not the path between, the work is done by a conservative force.
Case in point is the work done by gravity when an object is raised to a height $h$ above the ground. Since the force of gravity is always vertically downward, the dot product (and thus work done by gravity) is only non zero for the component of the displacement in the vertical direction, and zero for any horizontal deviations in the path.
Hope this helps.