Missing from the currently-accepted answer is the importance of steam.
If you use a damp towel to grab a heavy metal dish out of an oven at $\rm 350^\circ F = 180^\circ C$, the heat from the pan will enter the water in the towel much more rapidly than conduction can carry it to your hand. Instead some of the water in contact with the hot pan will instantly convert to steam, which is much more mobile than the water and (because of the pressure change involved in the vaporization) can move through any remaining air gaps in the towel very rapidly. When the steam reaches your cool hand it is very efficient at transferring the heat to you. Furthermore this rapid vaporization is the most dramatic in the parts of the damp towel that are held the most tightly to the hot pan --- which is exactly where your hand is.
Using a dry tea towel to grab a hot pan puts the heat into the towel fibers, which are poor conductors of heat, and which serve to separate your hand from the hot pan by a set of air gaps.
Using a damp or wet tea towel is a recipe for a ferocious steam burn.