To a first approximation, the scale is either balanced or not balanced, and remains so at all angles, so there is no formula that does what you're looking for. A simple model of a scale says that the angle should be either
- whatever angle you want if the weights on both sides are equal
- tipped all the way over to the heavier side if the weights are unequal
This assumes that the pans are equally-spaced from the fulcrum and the fulcrum is an idealized point and the scale is rigid and the masses are the same shape.
You can try this for yourself easily with an experiment I did just now. Balance a spoon on your finger. Put a penny in the bowl of the spoon. The spoon will tip all the way over, the penny will fall off, and the spoon falls off your finger. (It did for me at least - your results may vary depending on how ideal spoons and fingers are where you live.)
One way to modify this using a physics concept would be to move the fulcrum towards the heavier side. A scale is balanced when
$$L_1w_1 = L_2w_2$$
with $L_1$ and $L_2$ the length of the sides of the scale from the fulcrum and $w_1$ and $w_2$ the two weights. For example, if you put a 2 kg weight 10cm from the fulcrum to the right, it can be balanced by a 1 kg weight on the left if you put it 20cm away. This is the same principle a parent can use to play see-saw with their child. The physics term is a lever.
One way to see where this equation comes from is by balancing the torques exerted by the weights.
Another is to consider energy. The energy of the weights comes from the formula
$$U = w h$$
$U$ is the potential energy, $w$ is the weight, and $h$ is the height above ground. A basic principle of statics is that the scale will be balanced whenever this potential energy stays the same as you change the angle.
Suppose one arm is twice as long as the other. Then when you tilt the scale a little, that side will drop twice as far, so $h$ changes by twice as much for the long side as for the short side. That means that to have the scale balanced $w$ must be half as much for the long side. That way the change in $h$ is twice as much but $w$ is half at much, so when you multiply them to get the energy, you find the same thing for both sides.
For example, if with the 2kg and 1kg weights described above, the 1kg weight is on the long side. If it falls 1cm, then 2kg weight will rise 1/2 cm. Multiplying the change in height by the weight gives the same for both. (The weight is technically gravitational acceleration times the mass, so the weights are 10 Newtons and 20 Newtons because gravitational acceleration is 10 m/s^2.) This is true regardless of the angle, so the scale is either balanced or unbalanced at all angles.
To read some more, the scale is a simple example of a lever.