I understand that this is a consequence of the conservative nature of electric field, but then how does an electric dipole form closed loops?
Electric field lines of an electric dipole do not form closed loops .See here ,they begin at positive charges and end at negative charges.
But magnetic field lines form closed loops ,see the difference
Magnetic fields form closed loops because there are no magnetic monopoles.
See this post for more information
But unlike magnetic monopoles electric monopoles exist in nature so electric field lines need not form closed loops.