I thought of this question while waiting for a warm room to cool down
after opening a window.
A warm room is not just composed of warm air. The whole building has a huge thermal mass (basically, the heat capacity of walls, furniture, basement, ...) in comparison to the air inside it.
Your building is not perfectly sealed. Even when the windows are closed, the air change rate is close to $\frac{1}{\mathrm{h}}$. Which means that after 1 hour, the whole indoor air volume has been replaced by outside air. It does not mean that the indoor temperature is equal to the outside temperature after an hour, though.
If you open the windows, you might achieve an air change rate of $\approx \frac{20}{\mathrm{h}}$. Which means that after 3 minutes, you have replaced all the indoor air with fresh air. If you close the windows after 3 minutes, the indoor air temperature will almost revert back to the previous temperature, because the walls act as a thermal reservoir.
If walls have different temperatures (e.g. due to solar irradiance), convection will transfer heat much more efficiently than diffusion (the process you were asking for).
By the way, the best way to cool down a room in summer is:
- to avoid getting solar gains during the day, with external shading
- closing the windows during the day
- opening them wide all night long, in order to cool down the walls
- removing the external shading, if possible, during the night, in order to allow radiative cooling