I sometimes put a sweater on when I first get up on a cold morning. The sweater isn't so cold against my skin, but the zipper is. I get this. Even though they are both at the same temperature, the zipper is a better conductor of heat. Heat flows more easily from warm skin into the cold zipper, cooling the skin more effectively.
The reason a zipper is a better thermal conductor is that it is a metal. Electrons in the conduction band conduct heat as easily as electricity. On the other hand, wool is hair - an insulator.
When I get into a sleeping bag, it is cold like a zipper, even though the nylon and feathers are both insulators. Why is this?
I saw Stephan Bishof's answer to Could a sleeping bag be warmer if you are naked inside?. He says evaporation plays a role. I am not sure I buy this. It is cold even against dry skin, when you get static electricity from sliding in.
Is it simply that the sleeping bag is smooth and a sweater is fuzzy? Better contact makes better heat transport?
In summer you can avoid the whole by unpacking your bag when you first get to camp. It fills with warm air, and keeps it warm until night.