Imagine you have a single charge (not a dipole). We say it has zero potential energy at infinity. If we bring it close to another charge, it will end up with some non-zero PE:

$$V_+ = \frac{Q}{4\pi \epsilon_0 r}$$

Now if we have a second charge with opposite polarity, and we move it to the same distance $r$, it would have potential energy

$$V_- = -\frac{Q}{4\pi \epsilon_0 r}$$

The sum of these two is zero.

Now a dipole is really just the limit of two opposing charges brought infinitely close together while maintaining a constant product $p = Qd$ where $Q$ is the charge, and $d$ is the distance between them. The dipole vector points along the line joining the two charges.

Putting it all together, the distance $r$ for the two charges is the same if the dipole is at right angles; and so there is no net work done when you move a dipole from infinity to a distance $r$, and keep the dipole at right angles to the electric field.