The conduction band electrons in the metal, are called either(there are two main opinions on this site):
loosely bound to the atoms
Now the electrons move in zig zag way through the metal, as they interact with the atoms in the metal lattice. This constant interaction as they move in a zig zag way will give them their drift velocity, which is very slow.
Now you are asking why is this drift velocity not increasing as there is an external electric field present?
This external electric field is what gives these electrons kinetic energy, so they can move from one atom to another. If you remove this electric field, the electrons do not have the extra kinetic energy to keep moving, there is not electricity anymore.
You have to keep the electric field to keep giving the extra kinetic energy to the electrons to keep them moving through the lattice of the metal, interacting with the atoms.
The kinetic energy they gain from the electric field balances with the energy they lose as they interact with each atom. This is why the drift velocity is constant.