For forces that change along the way, displacement is not the thing to calculate work with. Let $\gamma : [0,1] \rightarrow \mathbb{R}^3$ be the (closed or open) path that the particle the force is exerted on follows. Then, the work done along that path is
$$ W[\gamma,F] = \oint_\gamma \vec{F}(\vec{x})\cdot \mathrm{d}\vec{x}$$
which is a line integral. If $\vec{F}$ is conservative, there is a function $V(\vec{x})$ such that $\nabla V(\vec{x}) = \vec{F}(\vec{x})$, then we can apply Stokes' theorem (or, less fancy, the fundamental theorem of calculus) to calculate the work by
$$\oint_\gamma \vec{F}(\vec{x})\cdot \mathrm{d}\vec{x} = \int_{\partial\gamma} V(\vec{x}) = V(\gamma(1)) - V(\gamma(0))$$
For closed paths, $\gamma(1) = \gamma(0)$, so this is zero. If there is no potential with $\nabla V = \vec{F}$, we cannot apply this argument and have to actually calculate the line integral, which may be anything, especially not zero.