Why would we freeze in space if there is no matter to conduct the heat away from us? If a near perfect vacuum is assumed to be in space, how can heat leak around from our body for example?
Although theoretically energy wishes to spread equally all around, it still needs a medium to drive it.
Or is it...?
 A: Radiant energy is the only type of heat transfer you can have when the pressure is very low.
Everything emits radiation and the major factors which control the rate of energy emission by radiation from a body is the temperature of the body, area of the body and the nature of the surface of the body.
The rate of emission is proportional tp the temperature in kelvin raised to the power four ($\propto T^4$).
So for a body there is a flow of radiation which depends on the temperature of the body $T_b$ and the temperature of the surroundings $T_s$.
If $T_b < T_s$ then there is a net gain of radiation by the body from the surroundings.
If $T_b = T_s$ then there is no loss or gain of radiation by the body.
If $T_b > T_s$ then there is a net loss of radiation from the body to the surroundings.   
When you are in space away from any other objects there is an input of energy from your surroundings (space) corresponding to radiation coming from a body at a temperature 2.7 K - the cosmic background radiation.
The human body is at a temperature of about 310 K so the ratio of energy emitted by the human body to that received from cosmic background radiation is approximately $\left ( \dfrac{310}{2.7}  \right)^4 \approx 1.7 \times 10^{8}$.
This will mean that you will lose radiant energy in space at quite a rate as explained in other comments.
As to what it would feel when in space without any protection try this when it is cold outside or when you you are near a freezer.
Standing inside the house put your hand near the inside, but not touching, a single glazed pane of glass or inside a freezer.
You will feel the cold.  In fact what you are feeling is the effect of being a net emitter of radiation.
A: In space where there is no matter to conduct heat, energy can still leave our body in the form of electromagnetic radiation. This will continue to happen until the amount of radiation emitted from our body and the amount of radiation from outer space absorbed by the body reach an equilibrium which would of course happen at a very low temperature. Thus a body would freeze in space provided that there's no source of significant radiation nearby.
A: To give a little color to the correct answer by @BruceLee and to expand on his question about the "color", a black body radiator of $2m^2$ area (which is approx. the area of a human body), emits approx. $900W$ in thermal radiation at the temperature of the human body. Since we only generate approx. $100W$ in heat at rest, we would very quickly lose body temperature and the skin would freeze very quickly, too.
Now, if we trust http://www.spectralcalc.com/blackbody_calculator/blackbody.php, then a $300K$ body of $2m^2$ would emit $2m^2\times 4.70255\times10^{-20} W/m^2sr\approx 1\times 10^{-19}W/sr$ or about one photon
 per second into the full angle in the spectral range of the visible light. That's kind of dark... very dark...and most of these photons would be near the deepest red that we can perceive. 
