I thought that if you're wet, a breeze will feel cold because the water moisture is evaporating and whisking away heat along with it. With a towel at room temperature (same temperature as a breeze), the heat will be whisked away even faster, all in a moment in fact. So shouldn't that feel really cold?
Evaporation cools you down, not water getting whisked away from your body!
Temperature has to do with the average energy states of all the particles in a substance. Only the high energy (think $212^o$ or higher) particles will escape via evaporation. If you take away all the high energy particles, what happens to the average? (hint it goes down).
On the other hand, with a towel, you're taking away any water molecules that touch the towel, so it doesn't change the average energy of the water on your skin and therefore it doesn't lower your skin temperature.
For a crude simulation, consider the distribution of temp. of water molecules on your skin (disclaimer: it's somewhat meaningless for an individual molecule to have a temperature, but the general idea works here):
The black line indicates the average temperature. Notice what happens to the average when you get rid of everything above 212:
And contrast with just taking away a bunch of water molecules at random with a towel:
The main flawed assumption here is that the towel carries away heat faster than evaporative cooling of the water. In reality, a dry or even slightly damp towel is a thermal insulator, meaning that rubbing yourself dry with it doesn't really transfer all that much heat from you to the towel. In the process, though, you're removing a very effective source of heat transfer (namely, evaporative cooling), so you get a net warming effect.
Supporting this line of thinking is the fact that rubbing a soaking wet towel on you is extremely unpleasant in its coldness, since you're not removing any of the water that's on you; you may in fact be prolonging the cooling by replenishing the water that evaporates.