I have made a simple sketch of how I think the system looks like. My problem is:
I always thought that the angle the balance makes is a function of the difference between the two masses (or the difference in torques): a small difference in mass will give a small angle, and in the limit the angle will be equal to zero when both masses are equal.
Now I cannot derive that from the picture. As I see it now, the torques will always be different, no matter at which angle the balance is tilted, since both torques scale with the same sine. This leads to a equilibrium only when the balance is completly vertical (or in my figure pressing against the triangle), which is contra-intuitive. Moreover, when the masses are the same, equilibrium would be obtained at every angle as can be seen from my figure (torques are equal independant of the angle of the balance), which would contradict with the working of a balance.
I see that my question kind of overlaps with this one, but the accepted answer doesn't solve my problem.
(One way to solve my problem, would be to draw a half circle instead of the triangle in my figure, so the lever of the torque becomes a function of the angle. But somehow in my head the schematic drawing of a balance is with a triangle in the middle. Is this perhaps the simple solution I'm looking for?)