I wouldn't say it is a myth. Like you say, it's complicated. When you are in the sun, the predominant source of heating is from incident radiation, whereas in the shade cooling would take place primarily through convection, conduction and evaporation. When you are in the shade, you are just not that hot compared to your surroundings for radiative cooling to be very effective (10 degree difference).
If you stay in the deep shade all summer, by all means, wear whatever colour you like. But don't go hiking with a black shirt on.
In the winter, where body temperature might be 45 degrees from ambient, radiation will play a big role. Of course, if you're well insulated then your overall surface temperature is close to ambient, and the effect may be negligible (e.g. a white parka and a black parka may perform similarly).