Suppose that the rod shown below in figure(1) is balanced. So the torque about point $O$ is $Mg.x-mg.y=0$.
[figure(1)]
Then imagine than M is lifted a little bit up making an angle of $\theta$ with the horizontal plane(figure(2)).
[figure(2)]
Then let's calculate the torque about point $O$.
Torque about point $O$ $$= Mg.x cos\theta - mg.y cos\theta $$ $$ =cos\theta(Mg.x-mg.y)$$ from the first equation:
$$=cos\theta.(0)$$ $$=0$$ Also we can find the torques about both ends of the rod are equal to zero. So shouldn't it be balanced in its new position?
And when talking about potential energy of each mass, when we lift one side up its P.E. is increased as well as the other one's P.E. is decreased. We can imagine that both of them cancel each other( This is obvious when $M=m$ and $x=y$).*
Thus shouldn't it be in neutral equilibrium?
But as we experience in day-to-day life balances always find a horizontal plane to be balanced. So what is wrong with my previous explanation?
Edit: The example of a beam scale may be irrelevant here. But I tried to balance such a rod and I couldn't keep it in neutral equilibrium. Is that because both masses are trying to keep their gravitational potential energy at a low level? Though this contrast again with above paragraph marked as *. Does it imply that these two masses do not act as a system?