Thermal balance
Thermal balance of a human body has been discussed, e.g., in this question: Stefan-Boltzmann law applied to the human body. There are a few points to note (some already brought up in other answers):
- Radiation is not the only mechanisms of heat loss (we need also to account for heat conduction and convection)
- Clothing plays important role in mitigating heat losses (or preventing overheating).
- The invisibility cloak is not necessarily equally transparent in both directions - i.e., it may contain the radiation emitted by the body, while not allowing the external radiation inside.
- Invisibility may apply to different spectral ranges: optical invisibility (like in fairy tales), invisibility to radars (like B2 bombers, which are well visible to human eye), invisibility to infrared cameras, etc.
Thermal balance without external radiation
For the ambient temperatures below the edge of the thermal neutral zone (see the discussion in the thread linked above) the heat balance depends on absorption of the ultraviolet radiation. If the invisibility cloak does not allow penetration of this radiation, one would indeed feel uncomfortably cold. However, as mentioned above, invisibility mainly implies the lack of visibility in radio-infrared-visible part of the spectrum, i.e., any realistic invisibility cloak is likely to allow the ultraviolet radiation.
On the other hand, in conditions above the upper critical temperature of the thermal neutrality zone, one would rather be shielded from the external radiation, in order to avoid overheating, i.e., the discussion in the previous paragraph needs to be reversed mutatis mutandis.
Finally, these effects are minimized when one is shielded from the sunlight or during the night time.

Update
I think that proper discussion of this question requires two key definitions:
- what is to be invisible?
- what is to feel cold?
Feeling cold
Feeling cold refers to the temperature balance, which is discussed at length in the post linked above and the links therein. It means balance between the heat obtained from the internal metabolism (digesting food) and radiation absorbed vs. the heat lost via radiation, convection and heat conduction. This is the theme developed in the maina nswer.
Invisibility
Invisible means not being seen by eyes or a detector. It is therefore not absolute - it depends on the range of the electromagnetic field detected and on the method of detection. Onecan distinguish at least three situations:
Infrared: Radiation emitted by the body
Infrared cameras typically detect the radiation emitted by the body, which is hotter than the environment. Being invisible to infrared cameras means reducing the emission of the radiation this can be achieved by retaining heat inside, or reemitting it in different frequency range, or making sure that it is quickly dissipated (making the body blurred rather than invisible).
Optical: radiation scattered by the body
In conventional sense invisibility means not being seen by eyes or photo camera. In this case the light is the scattered one - e.g., the light of the Sun. The invisibility can then be achieved by making the object transparent, or mimicing the transparency via ingenious waveguiding or holographic systems. Again, we can talk about making the object blurred rather than truly invisible by fiddling with how the light is scattered.
Radio: active location
Another frequent use of invisibility is in the context of stealth airplanes. In this case the method of detecting is mainly the radar, which sends radio waves that are reflected by the airplane. Being invisible to the radar then means reducing the intensity of waves reflected in the direction of the radar or simply absorbing this radiation - in optical sense the object then appears black rather than invisible.