This might be a very elementary question. Considering that electrons in typical metals have very high Fermi velocity, why is it that when you touch a metal you feel cold? My intuition tells me that due to Fermi-Dirac distribution only a tiny fraction of electrons near the Fermi surface participate in heat conduction. Heat transfer due to electrons is tiny compared to transfer due to phonons. Is this reasoning correct?
In a Fermi-Dirac distribution, the relationship between temperature and the speed of particles is not intuitive. Even at cold temperatures, fermions can have high speeds simply because of degeneracy - the lower momentum states "fill up", leaving only states with large momentum available, and this is true even at very cold temperatures. However, the heat capacity of the conduction electrons is negligible - heat cannot be extracted precisely because there are no lower energy states available.
The reason that metals feel cold is because they have a high thermal conductivity. This can also be attributed to degeneracy of the conduction electrons, since in a degenerate electron gas there are few available lower momentum slots into which a conduction electron can be scattered. This means that the electrons have a relatively long mean free path between scattering events and are able to transfer heat efficiently from your finger into the metal and then away. Thus the temperature of the metal where you touch it does not rise to match your skin temperature.
I don't think phonons come into this at all. In metals, electron heat conduction dominates phonon transport.
If you think about the infinite square well problem, the states with higher energy have higher momentum, (and also a higher velocity). However, it is better to think of the higher energy states as higher frequency standing waves. Because they have a higher frequency, the have to "travel faster", which is where the large velocity comes from in the Fermi velocity.
However, this does not have an affect on the temperature, because this is not classical, and one cannot use the equipartition of energy in this case. I should also add that if one were to touch a metal and have it be "hot", metals would never be in thermal equilibrium with the environment thereby violating the second law of thermodynamics.
As for the second part of your question, your intuition is correct. At room temperature, in fact only the electrons near the Fermi energy participate in heat transport. Phonons also do participate in heat transport, but it is not always obvious which one contributes more. Electrons do their fair share of transport in many metals.