To understand some of the weirder properties of metals it is probably necessary to think about Fermi surfaces pretty deeply.
When you think about applying an electric field you are disturbing the way the electrons are distributed at or near the Fermi surface. For simple cases that seems pretty straightforward and and you can think about fermi gas and relaxation times etc. If you think of the simple case of a Fermi sphere you can think of the Fermi sphere shifting and little bit when the field is applied and the amount of that shift is related to the drift velocity.
When you have a crystal lattice, rather than having a sphere you can have more complicated shapes for the Fermi surface. This depends on crystal lattice and is related to the band structure.
So now think about applying a magnetic field. An electron in a magnetic field moving at a velocity in a vacuum would move in a circle and not do any work. In the crystal with the magnetic field the same kind of thing happens but the electron stays on the Fermi surface and depending on the surface can have an electron like orbit, or could have a hole like orbit or move from Brillioun zone to Brillioun zone in an open orbit.
So for copper which looks kind of like a sphere with bumps you have electron like orbits and the Hall effect behaves like you would expect with the electrons being deflected by the magnetic field (Lorentz force) but in aluminum you can find that the Hall effect has the wrong sign (anomalous Hall effect) because the Fermi surface has the hole like orbits.
But in both cases as you apply the fields the interesting stuff is at the Fermi surface where and electron can move to a nearby unoccupied state.