In my book they say work done is zero because force and displacement are mutually perpendicular.
Trust me, it doesn't say that.
When you're moving the object to the left or right (along the $x$-axis), you need to apply a net force that is parallel to the displacement vector. Work is then done acc.:
$$W=\int_0^xFdx$$
But as you're not moving the object along the $y$-axis, the force to hold it steady in the vertical sense does not perform any work because that force is perpendicular to the $x$ displacement.
Edit:
I am still unable to figure out whether I need to apply some force in the direction of motion or not. If not then how does the object decides to move towards right or left?
This is explained by Newton's laws of motion.
First law: an object either remains at rest or continues to move at a constant velocity, unless acted upon by a net force.
Initially you're holding the object still, both in the vertical sense ($y$) and horizontal sense ($x$).
In the vertical sense, gravity $F=mg$ acts on it but you counteract that with an equal but opposite force, hence there is no net force acting on it in the $y$ direction.
In the $x$ direction there's no motion either so by Newton this means that the net force in the $x$-direction is also zero. Now to make the object move in the $x$-direction you have to apply a net force to it (in the direction of desired motion), your arm has to 'pull' or 'push' on it.
To illustrate this we use a free body diagram:
Left: the object is motionless and $F_1$ is the force you provide to counbteract gravity.
Right: $F_2$ is the 'pulling' or 'pushing' force, depending on its direction the object will move left or right.
Newton's second law basically quantifies all this and allows to calculate the actual accelerations.