I want to understand why the potential energy of an ideal magnetic dipole with dipole moment $\boldsymbol{m}$ in a non-uniform magnetic field $\boldsymbol{B}$ (neglecting the term to keep the magnitude of the dipole moment fixed) is: \begin{equation} U=-\boldsymbol{m}\cdot\boldsymbol{B} \end{equation}
The force on the dipole is: \begin{equation} \boldsymbol{F}=\nabla(\boldsymbol{m}\cdot\boldsymbol{B}) \end{equation}
The torque is: \begin{equation} \boldsymbol{\tau}=\boldsymbol{m}\times\boldsymbol{B} \end{equation}
The derivations I found either compute (using the gradient theorem)
\begin{equation} U=-\int_{\infty}^\boldsymbol{r}\boldsymbol{F}\cdot d\boldsymbol{l} \end{equation}
with assumptions on the field at infinity, or they use torque:
\begin{equation} U=-\int_{\theta_0}^{\theta}\tau d\theta \end{equation}
that leads to the same result (except and additive constant).
The question is: why are these used separately? In terms of work, we must consider the work to bring the dipole from infinity (integrate the force) and then the work to rotate the dipole (integrate the torque), so what leads to this? Maybe I am considering two different energies? Because if the field is uniform the force is zero (thus zero work moving the dipole), but the torque is not, and we get that same energy from the torque.