EDITED TO ADDThere are two points relevant for the discussion: air itself carries a very small amount of thermal energy and it is a very poor thermal conductor.
For the first point, I think it is interesting to consider the product $\text{density} \times \text{specific heat}$, that is the amount of energy per unit volume that can be transferred for every $\text{K}$ of temperature difference. As of order of magnitudes, the specific heat is roughly comparable, but the density of air is Zephyr stated$10^3$ times smaller than the density of a common metal; this means that for a given volume there are much less "molecules" of air that can store thermal energy than in his commenta solid metal, the point hereand hence air has much less thermal energy and it is not enough to cause you a dangerous rise of the temperature.
The rate at which energy is transferred to your hand, that is the flow of heat from the other objects (air included) to your hand. In the same amount of time and exposed surface, touching air or a solid object causes you get a very different amount of energy transferred to you. The relevant quantity to consider is thermal conductivity, that is the energy transferred per unit time, surface and temperature difference. I added this to give more visibility to his comment; my original answer follows.
Air is a very poor conductor of heat, the reason being the fact that the molecules are less concentrated and less interacting with each other, as you conjectured (this is not very precise, but in general situations this way of thinking works).
On the opposite, solids are in general better conductors: this is the reason why you should not touch anything inside the oven.
Considering order of magnitudes, according to Wikipedia, air has a thermal conductivity $ \lesssim 10^{-1} \ \text{W/(m K)} $, whereas for metals is higher at least of two orders of magnitude.
I really thank Zephyr and Chemical Engineer for the insight that they brought to my original answer, that was much poorer but got an unexpected fame.