That is, why is the potential energy with the orbitals overlapping less than with the Hydrogen atoms 'independent'. Similarly, why is a noble gas configuration stabler than if an electron were to be removed or added? Is this because pairs of electrons are more stable than single electrons? If so, why?
The pairs of electrons are more stable if you see the pair as a filled orbital (which can contain maximum of 2 different spin electrons via Pauli's Exclusion Principle).
I think the simplest way of explaining the lower energy of two-nuclei orbital is that the size of the electron's 'playground' grows 2 times, and as the area over which the electron may soar increases, it's kinetic energy decreases.