Valence electrons are outermost (closest to zero) in energy on a binding energy scale. Valence electrons in atoms form the covalent, ionic, or metallic bonds in molecules or bulk substances. In a single covalent bond between two atoms, valence electrons that make (two) molecular orbitals are shared between the two atoms. In metallic bonds, valence electrons are shared between all atoms in the metal.
For bonding in metals, at absolute zero, all electrons are in bound states in the equivalent of a highest occupied molecular orbital (HOMO) band. The topmost occupied level in the HOMO band defines what is called the Fermi energy. The occupied portion of the HOMO band below the Fermi energy is called the valence band. The bound states in the valence band are shared by all atoms collectively. The electrons in the valence band are said to be delocalized throughout the valence band. They have no specific position that defines their location. By comparison, the electrons in metals at higher binding core level energy states are spatially localized about individual atoms. This is not about the depth of the potential well on any given atom, rather it is that core level electrons are at binding energies that are factors of 10x or more above binding energies for valence electrons. Core level electrons are essentially entirely localized about the metal atom. Since valence electrons are delocalized, they have no sense of a specific place to go when we apply a voltage field--essentially, they are already "everywhere" spatially. Since valence electrons are bound, they can only be displaced by a voltage field, they cannot be moved out of position. Think of voltage field shifting an entire delocalized valence band cloud of electrons rather than of a voltage field causing any one or more specific electrons to travel from point A to point B. The former is polarization (leading to capacitance), the latter is current (leading to conduction).
Any free (unoccupied) levels above the Fermi energy constitute what is called the conduction band. No electrons occupy these states at absolute zero. No electrons are "free" from their bound, valence band states; they are all below the Fermi energy. At absolute zero in temperature, since no electrons are free to travel through the conduction band, metals are electrical insulators.
As we raise temperature above absolute zero, electrons can move above the Fermi energy. This is a thermal excitation process; it is not induced by voltage. Electrons in the conduction band region are less delocalized, and, because they are unbound in the shared metallic bonds, they are also free to travel anywhere in the metal. Specifically, they are free to carry current when we apply a voltage field.
The picture is not about whether electrons are tightly or loosely bound. The distinction is about whether electrons are bound and delocalized in the valence band or whether they are unbound and more localized in the conduction band. The former cannot carry current; they latter can respond to a voltage field and travel (carry current) from one location to another.
An insulator is nothing more than a system where the portion that we call the valence band is completely full and the portion that we call the conduction band is separated by an energy gap. The conduction band sits closer to the zero point of energy while the valence band lies below it at higher binding energy. This is also the distinction between a metal and a semiconductor; the former has no band gap, while the latter has a band gap. The band gap in semiconductors is smaller than that in insulators. Electrons from the (full) valence band can be thermally excited to hop over the band gap in semiconductors. Those electrons that reach the conduction band in semiconductors can carry current (the details about holes in semiconductors are the subject of a different discussion). The band gap in insulators is too large for thermal excitation to cause any electrons from the (delocalized yet bound) valence band states to reach the (localized yet unbound) conduction band states.
Finally, metals behave as insulators at absolute zero in temperature because all electrons are in the bound states in the valence band. Even though phonons are frozen out and therefore resistance becomes zero, no free electrons occupy the available states to carry current.
All told, for the same temperature in a given volume of a solid, insulators have no "free" electrons (electrons in the conduction band) because the band gap is too large, semiconductors have some "free" electrons because the band gap exists but can be overcome by thermal motion, and metals have more "free" electrons because no band gap exists.
To close on one note in your question, polarization is the displacement of charge remaining tied to a fixed point, conduction is the movement of charge from point A to point B. When we relax a voltage field, charges displaced by polarization return back to the fixed point, but charges that have moved to point B by conduction do not return back to point A. Charges that cause polarization do not contribute to current, and charges that cause current do not contribute to polarization.