If you were invisible, would you also be cold? If you were invisible, would you also be cold? (Since light passes through you, so should thermal radiation.)
Additionally, I'd like to know if you were wearing invisible clothes, would they keep you warm? In my understanding, the heat radiation from the body would pass through the cloth.
Is it even necessary to be permeable for heat radiation in order to be invisible? Could there be a form of invisibility (hypothetically speaking, of course) that makes you permeable for light in the visible spectrum, but not for heat radiation? Can those two things be separated?
 A: Thermal balance
Thermal balance of a human body has been discussed, e.g., in this question: Stefan-Boltzmann law applied to the human body. There are a few points to note (some already brought up in other answers):

*

*Radiation is not the only mechanisms of heat loss (we need also to account for heat conduction and convection)

*Clothing plays important role in mitigating heat losses (or preventing overheating).

*The invisibility cloak is not necessarily equally transparent in both directions - i.e., it may contain the radiation emitted by the body, while not allowing the external radiation inside.

*Invisibility may apply to different spectral ranges: optical invisibility (like in fairy tales), invisibility to radars (like B2 bombers, which are well visible to human eye), invisibility to infrared cameras, etc.

Thermal balance without external radiation
For the ambient temperatures below the edge of the thermal neutral zone (see the discussion in the thread linked above) the heat balance depends on absorption of the ultraviolet radiation. If the invisibility cloak does not allow penetration of this radiation, one would indeed feel uncomfortably cold. However, as mentioned above, invisibility mainly implies the lack of visibility in radio-infrared-visible part of the spectrum, i.e., any realistic invisibility cloak is likely to allow the ultraviolet radiation.
On the other hand, in conditions above the upper critical temperature of the thermal neutrality zone, one would rather be shielded from the external radiation, in order to avoid overheating, i.e., the discussion in the previous paragraph needs to be reversed mutatis mutandis.
Finally, these effects are minimized when one is shielded from the sunlight or during the night time.

Update
I think that proper discussion of this question requires two key definitions:

*

*what is to be invisible?

*what is to feel cold?

Feeling cold
Feeling cold refers to the temperature balance, which is discussed at length in the post linked above and the links therein. It means balance between the heat obtained from the internal metabolism (digesting food) and radiation absorbed vs. the heat lost via radiation, convection and heat conduction. This is the theme developed in the maina nswer.
Invisibility
Invisible means not being seen by eyes or a detector. It is therefore not absolute - it depends on the range of the electromagnetic field detected and on the method of detection. Onecan distinguish at least three situations:
Infrared: Radiation emitted by the body
Infrared cameras typically detect the radiation emitted by the body, which is hotter than the environment. Being invisible to infrared cameras means reducing the emission of the radiation this can be achieved by retaining heat inside, or reemitting it in different frequency range, or making sure that it is quickly dissipated (making the body blurred rather than invisible).
Optical: radiation scattered by the body
In conventional sense invisibility means not being seen by eyes or photo camera. In this case the light is the scattered one - e.g., the light of the Sun. The invisibility can then be achieved by making the object transparent, or mimicing the transparency via ingenious waveguiding or holographic systems. Again, we can talk about making the object blurred rather than truly invisible by fiddling with how the light is scattered.
Radio: active location
Another frequent use of invisibility is in the context of stealth airplanes. In this case the method of detecting is mainly the radar, which sends radio waves that are reflected by the airplane. Being invisible to the radar then means reducing the intensity of waves reflected in the direction of the radar or simply absorbing this radiation - in optical sense the object then appears black rather than invisible.
A: Also, don't forget that not all heat comes to our bodies from outside. We convert a fair amount of sugar into heat.
A: As most of us know from experience, radiation is not a major contributor to the body's heat gain or loss. Unless you're sitting in the sun, or in front of a good fire, radiative heat transfer is almost undetectable by the senses.  Convection and conduction are the major methods of heat transfer.
You also need to remember that the body is a net heat producer.  The metabolism of food produces heat.  If we exercise, or shiver, we produce extra heat. We wear clothing, and/or perspire, to regulate body temperature.  Clothing is almost entirely* concerned with regulating convection & conduction.
Perhaps a couple of real-world examples would help in understanding.
First, when I cross-country ski (an exercise that burns a lot of calories if done right) on a calm day, I can work up a sweat while wearing nothing but a light shirt and sweat pants.  Stop for a while, and I'll need to haul out the down vest to keep warm.  So radiation is not a factor here.
Second, when I go to the beach, I wear very little clothing, yet I don't become cold because my body heat is radiating away.  If I go in the water, I may get cold because the water greatly increases convective heat loss.
So the bottom line is that if your invisible clothing was sufficient to keep you warm before it (and you) became invisible, it will still keep you warm when you are invisible.
*There's also perspiration,which I won't get into beyond saying that's why experienced people dress in layers when exercising outdoors in the winter.
A: Not necessarily. Glass is transparent and it can be heated with convection, conduction and some radiation. It will maintain any temperature around it.
A: Although the premise of this question is, well, questionable, I’ll answer because invisibility cloaks do exist. Of course, as of now, they work imperfectly, in impractical ways, and for only a narrow spectrum of light. And this last part is the key. Just because something bends light of one color does not mean it will bend light of another. When one colloquially discusses invisibility, visible light is implied; however, thermal radiation is primarily infrared. So if you’re standing at the center of an invisibility cloak, it will probably still radiate warmth.
And, of course, radiation is not the only way to transfer heat. There’s also conduction and convection, so let’s not forget those.
A: Warm dark matter is not what you may think, but rather a hypothetical component of dark matter, which may consist of, e.g., sterile neutrinos.
The point here is that temperature is essentially the average kinetic energy (at least for a gas), and it does not fundamentally matter whether the particles interacat with the electromagnetic field. Hence, a gas of neutrinos has a well-defined temperature, despite being invisible (to all wavelengths).
Note that while neutrinos and similar particles interact very little and so would probably not form molecules or complex structures, there not fundamental obstacle to imagine another sector of matter which has significant interactions in itself, but is decoupled from standard model matter.
A: Two key points to remember:

*

*Radiation is not the only form of heat transfer. There's also conduction and convection.

*Being invisible doesn't only mean that you shouldn't absorb any light. It also means you shouldn't emit any.

From the perspective of heat transfer, making yourself invisible would be similar to wearing a skintight body suit made of reflective mylar (like emergency blankets).
You could definitely be kept warm by invisible clothes. Think about how the inside of a greenhouse stays warm.
The entire EM spectrum carries heat, but if by "heat radiation" you just mean infrared, then yes, you can absorb or reflect it while letting visible light pass.
A: 
if you were wearing invisible clothes, would they keep you warm? In my understanding, the heat radiation from the body would pass through the cloth.

Clothes don't just block the radiated heat, they also stop conduction and convection with the ambient air. I haven't done the calculations but I suspect invisible clothes would only be marginally less warm.
A: Thermoregulation

There are four avenues of heat loss: evaporation, convection, conduction, and radiation. If skin temperature is greater than that of the surrounding air temperature, the body can lose heat by convection and conduction. But, if air temperature of the surroundings is greater than that of the skin, the body gains heat by convection and conduction. In such conditions, the only means by which the body can rid itself of heat is by evaporation.


So, when the surrounding temperature is higher than the skin temperature, anything that prevents adequate evaporation will cause the internal body temperature to rise.[24] During intense physical activity (e.g. sports), evaporation becomes the main avenue of heat loss.[25] Humidity affects thermoregulation by limiting sweat evaporation and thus heat loss.[26]

