Most materials you use in everyday life contain far more moisture than you might believe. This is a major reason materials meant to be exposed to space are specially designed and tested. In a general vacuum, most fabrics and many plastic will outgas - all of the absorbed moisture and oils will work their way to the surface and boil off - which is a major source of contamination for sensitive equipment. With a wet shirt in a general vacuum, the water would boil away extremely quickly; as would the oils and even some of the lower quality waxes (depending on the material).
When behind Earth, the temperature does drop significantly. However, because the shirt has its initial heat from the ground and because the only way of losing that heat in space is through radiative and evaporative heat loss, the water on the shirt will explosively boil off almost instantly. The small water drops that might fly off the shirt will freeze to ice pellets lateralmost immediately, but the shirt will be more or less dry.
With the Sun shining on it, the shirt will again "insta-dry" but all of the water will vaporize; there will not be small ice pellets flying away from the shirt.
Overall though, I don't recommend doing it. The moisture and oils that outgas from your clothes will put large stresses on the fabrics and might ruin your clothes. Also, you could shrink your new sweater and that was a gift from your grandmother; you wouldn't want that right?