What happened to high temperatures in the universe? We know that after the big bang the temperature was about 10^32 K.
But now the average temperature of the universe is about 4 K. What happened to the temperature at that time?  Where did it go?
 A: One has to first understand  temperature. It is a measure of the heat in a body . Suppose you pour a glass of boiling water in a bucket of icedwater. What happened to the temperature of that volume of water? The volume of the boiling water was diluted into the cold water, and the cold water became a bit warmer.
An easy connection of temperature with the *average energy of the molecules comprising the material can be found in the statistical mechanics  definition of temperature:

Where E_k is the kinetic energy of each molecule and the barred symbol the average of the ensemble. 
Here is how the monatomic gas would look if we could see such magnifications:


The temperature of an ideal monatomic gas is related to the average kinetic energy of its atoms. In this animation, the size of helium atoms relative to their spacing is shown to scale under 1950 atmospheres of pressure. These atoms have a certain, average speed (slowed down here two trillion times from room temperature).

Here is a brief history of temperature after the Big Bang.

and a web page  on the temperature history of the universe.
The physical mechanism of the cooling is  used in our refrigerators and air conditioners:

As a gas expands, the average distance between molecules grows. Because of intermolecular attractive forces (see Van der Waals force), expansion causes an increase in the potential energy of the gas. If no external work is extracted in the process and no heat is transferred, the total energy of the gas remains the same because of the conservation of energy. The increase in potential energy thus implies a decrease in kinetic energy and therefore in temperature.

As pointed out in the comment the extrapolation to the expanding universe is not that simple. The primordial soup is more complicated than a simple cooling engine as a lot of the energy is taken away by electromagnetic radiation . Radiation is studied through the black body spectrum.  The temperature of a photon gas is derived here. The concept that the volume element expands as the universe expands while the number of photons remains constant leads mathematically to a lower temperature for the photon gas too. Thus the original energy exists in an expanded volume which has a lower black body temperature.
A: Maybe it is simplistic but look at it like this:
Think of temperature as the height of a water resevoir. The water represents is heat energy and as the resevoir base expands and becomes larger, the water level drops, whilst retaining the same amount of water (energy).
Remember temperature is a statistical measurement, and not an inrinsic quantity to matter. A single atom does not have an temperature, but an average speed which we relate to temperature. (Look up the kinetic theory of gases for more into this).
A: During that time the universe expanded, so energy is then spread out as it expanded.
It then gets spread out so much that it is 4K now.
