There are a couple reasons.
A big one is obviously the difference between heat and energy, as MaDrung already talked about quite a bit.
If you're familiar with electric circuits, I will try and expand upon temperature and heat transfer with a circuit analogy. In a circuit the current flows across a potential difference given by voltage. The current flowing depends on voltage difference and the resistance.
For the heating analogy; voltage is the same as temperature, and heat transfer is the same as current. The resistance to heat transfer depends on the real situation in the heat transfer medium. If two objects are at the same temperature; there should be no heat transfer between them. The amount of heat transfer is proportional to temperature, and is also inverse proportional to resistance.
When our bodies feel hot or cold, they aren't actually feeling the temperature. They are sensing the heat gain or heat loss. The heat loss depends on more than just temperature; it also depends on that "resistance". The resistance is changed by many things including how many layers of clothes you have on, how windy it is, and how humid the air is. This is one reason why two places with the same temperature may feel different.
Another big reason is the climate you live in and human psychology. If your average temperature is warm year round (like Australia) then 15 C isn't really a nice day (this also depends on seasons). On the other hand in England, 15 C could be a reasonable temperature, especially if it was near winter; then it could be an unusually warm day. It's like going down south in the winter for vacation. It seems really warm to you; but it's regular weather for the locals.