Imagine the "coil is totally free" to move on a flat surface the plane of which is perpendicular to the external magnetic field.
The torque and forces acting on the coil will make the coil orientate itself so that the magnetic field that is produced inside the coil is in the same direction as the external magnetic field as this is the lowest potential energy state.
The "plane" of the coil is now at right angles to the external magnetic field and so the forces on the coil will be outwards.
These forces will thus make the coil form a circle to reach a stable equilibrium state with the net force and the net torque on the coil both zero.
U will be minimum when MBcos$\alpha$ will be maximum. For this:-
- Cos $\alpha$ should be maximum (=1) which is possible when $\alpha$ is 0°,i.e, the angle between B and M is 0°
- When M is maximum, for this:-
we know M=NIA, where N is the no. Of turns, I is the current flowing and A is the area of the loop.
Now I and N are constant. So only the area can change and A should be maximum. And we know that a circle has the maximum are for a given perimeter. So the coil changes into a circular shape.
Now coming to your second question:
If the coil found itself with its magnetic field in the opposite direction to the external magnetic field and was unable to flip over then the forces would be inwards and the coil would collapse in on itself as this would be a minimum potential energy state for this configuration.
In the above figure, if the loop is not allowed to flip, it will in fact collapse to attain the minimum potential energy in this situation.