Some physicists you all are! The relevant issue is that the clothes you wear (unlike the down or synthetic fibers of the bag) hold moisture next to your body, changing both the conductivity of the medium touching your skin and the evaporative heat loss dynamics of the body-bag system. So, physicists, please discuss the effects of creating a moist fabric layer around the body vs leaving the body naked to freely transpire and the vapor to move more efficiently out of the bag.

The relevant issue is that the clothes you wear (unlike the down or synthetic fibers of the bag) hold moisture next to your body. Changing both the thermal conductivity of the medium surrounding your skin allows energy transport to the environment. The evaporative heat loss dynamics of the body-bag system is described by another thermal conductivity.

I propose the effects of creating a moist fabric layer allows a more efficient heat flow from the body than an air insulator. Water transport through your fabric should be best in your sleeping bag, naked a should feel colder and you will freeze in wet cotton cloths.