I understand that the torque acting on a magnetic dipole in a magnetic field is given by $$\vec\tau=\vec\mu\times\vec B \therefore \tau = \mu B sin\phi$$ It follows that the work done as the dipole moves through an infinitesimal angular displacement is $$dW=\tau d \phi \therefore W=\mu B\int sin\phi d\phi = -\mu Bcos\phi \therefore W=-\vec\mu\cdot \vec B$$ Where I have become confused is that the potential energy of a magnetic dipole in a magnetic field is given by the same expression: $U=-\vec\mu\cdot\vec B $. So, if the work done on a magnetic dipole through an angular displacement is given by the negative expression above, shouldn't the expression for potential energy then be: $U=\vec\mu\cdot\vec B$ rather than the negative of the same expression?
The only thing I can think of is if the torque is caused by an external force then the change in potential energy is equal to the work done by the external torque but I have a feeling this is not the correct way of looking at it.
Any help would be greatly appreciated!