Mathematically you can surely see that $\vec{J}=0$ outside the wire and therefore the differential form of Ampere's law demands that $\nabla \times \vec{B} = 0$.
I guess what is confusing is that curl is often described in terms of "curling field lines", but that isn't entirely accurate - straight field lines can have a curl and curved field lines can have zero curl (as in this case).
A better analogy is to consider the insertion of a small paddle wheel into the problem and imagine the field lines represent a fluid flow. The question you have to ask is whether the paddle wheel will rotate.
So in this case, what would happen is the paddle wheel would not rotate because the field is curl-free. See https://physics.stackexchange.com/a/302883/43351