IR is an extremely broad wavelength range (from near infrared, 800 nm to far infrared up to 1000000 nm). What you deal with when you experiment with a TV remote and a phone camera is near infrared (800-1000 nm). The material properties in that range are still pretty comparable to the optical range (although we don't see "infrared colors"). With a near-infrared filtering camera you will be surprised to see that many colored objects appear white, though. This is especially striking for many black colored objects. The reason is probably simply that the designed objects are not "optimized" for the NIR range because nobody sees these "colors" anyway.
However, this has nothing to do with heat radiation (or is rather only a tiny part of it) which starts at about 3000 nm up to 50000 nm. Absorption spectra in that range are totally unrelated to the near infrared. The latter is still dominated by electronic transitions (similar to the colors we see) inside molecules, while the latter is dominated by the vibrations between the molecules.
That is the reason why you cannot tell anything about the thermal radiation behavior (especially reflectivity, or "FIR color" so to say) of clothes with various visible colors. It's totally different physics, so to say. The only notable exception are metals: they reflect well in the optical and NIR range, and so do they in the FIR range. This is the reason why it is very difficult to determine the temperature of metal objects with a thermal camera (you might just see the reflections of nearby objects, e.g. yourself, and hence, their temperatures instead of the true temperature of the metal part).
What does have an indirect thermal effect, though, is the absorption of visible light and NIR and the subsequent conversion into heat radiation (much larger wavelength) due the temperature of the object being raised. Black clothes absorb the light of the sun very effectively and raise your temperature a lot while you are in the sun, as opposed to white clothes.
I would claim that what you have read in your book is a very popular science myth, set up to explain the confusing fact that people in the middle-east or africa wear black clothes, where this is actually probably just an accidental cultural mannerism, rather than a kind of "intuitive science by wise ancient grey-bearded sufi shamans".