This article on [triboelectricity][1] gives a triboelectric series, listing materials according to whether they will expel electrons or acquire them , when rubbed or in contact.
Towel is not in there :) but cotton is, with 0 but it is more positive than rubber. So the answer is that the electrons will move to the balloon, if you manage to get cotton to give up its electrons :) . Did you do the experiment? A woolen towel would be much better. Maybe it was your hand that gave up the electrons after all.
This is what wikipedia has to say on the causes:
After coming into contact, a chemical bond is formed between some parts of the two surfaces, called adhesion, and charges move from one material to the other to equalize their electrochemical potential. This is what creates the net charge imbalance between the objects. When separated, some of the bonded atoms have a tendency to keep extra electrons, and some a tendency to give them away, though the imbalance will be partially destroyed by tunneling or electrical breakdown (usually corona discharge). In addition, some materials may exchange ions of differing mobility, or exchange charged fragments of larger molecules.
Hence the need of a table. [1]: http://en.wikipedia.org/wiki/Triboelectric_series