Throughout this whole question, I will be referring solely to single element solids. According to band theory, ns and np bands are close enough in energy to overlap and create one band with 8N states available to fill. This band then splits into two 4N bands when the interatomic distance between atoms in the solid increases.
This model explains why diamond is an insulator. A single carbon atom has four valence electrons, so N carbon atoms will have 4N of them. These electrons fill the lower of the 4N bands (in this case, the valence band) completely, meaning that diamond cannot conduct electricity at normal conditions.
However, I don't understand why elements such as boron, nitrogen, oxygen and fluorine aren't able to. In boron, there would be 3N filled states with N empty states still remaining. In nitrogen, there would be 5N states and the higher of the 4N bands would start to fill, leaving another 3N empty states for conduction. The same goes for all non-metallic elements which aren't in the group 14 or 18. Why, then, are these regarded as semi-conductors or insulators even though they have unfilled states in their valence bands, which is a condition for conductivity?