It sounds like you know some of the most important summary points about blackbody radiation, but here is a reference on the subject, since I will be talking almost entirely about blackbody radiation: https://en.wikipedia.org/wiki/Black-body_radiation
Given any temperature, there is a certain emission spectrum (see https://en.wikipedia.org/wiki/Planck%27s_law) describing the mixture of photons which a body at that temperature emits. As you already noted, the peak frequency of this spectrum increases as the temperature increases. It is also of interest to note that an increase in temperature increases photon emission at ALL frequencies, not just higher frequencies.
The short form answer to your question is that we live in on Earth, where temperatures tend to be in the 200-400K range. Even a campfire doesn't make it much above 1500K. At all of these temperatures (yes, including the campfire) the VAST majority of the energy radiated is in the infrared range. If you put a filter between yourself and a campfire which absorbed all visible light and transmitted all infrared light, you would feel just as warm. So it is natural for us to associate infrared radiation with heat. The sun is the only everyday example of something that warms us noticeably with visible light, and sunlight already holds its own unique place in the human experience. Sunlight feels warm. A physicist can get out sensitive instruments and observe that in fact all light warms us slightly, but as far as what we can feel with our own nerves goes, it is only infrared and sunlight that seem warm.
If we were plasma beings that inhabited the core of the sun, perhaps we would associate visible light (or some energetic subatomic particle or other...) with the transmission of thermal energy, but we aren't and we don't.
In the end, all light transmits energy, and heat is just energy in the form of atoms exercising their degrees of freedom. So there is no clear cut distinction between the way infrared radiation interacts with heat and the way any other radiation does. But over most of the wide range of commonly studied environments, heat is mostly ratiaded as infrared photons. Thus the association.
Regarding your thought about incandescent lights vs more efficient alternatives, we replaced our 60-100 Watt bulbs with 7-20 Watt bulbs, so they really don't warm us up anywhere near as much as the old ones. If we had replaced the bulbs with equivalent wattages, then your thought would be correct, but we would be blinded by our lamps!