> In my book they say work done is zero because force and displacement are mutually perpendicular. 

Trust me, it doesn't say that.

When you're moving the object to the left or right (along the $x$-axis), you need to apply a net force that is *parallel* to the displacement vector. Work is then done acc.:

$$W=\int_0^xFdx$$

But as you're not moving the object along the $y$-axis, the force to hold it steady in the vertical sense does not perform any work because that force **is** perpendicular to the $x$ displacement.