This is the trick of a counter flow heat exchanger, the air streams arent in thermal contact when both are at 20°C.
In a parallel flow, 20°C would be the temperature both streams approach as they exchange energy, and the rate of heat exchange decreases as they get closer to the same temperature. This is basically the same as mixing all the air, except you could keep the supply and exhaust separated.
The thing is, we dont want to mix the temperatures, we want to switch them around. So instead, the coldest outside air is in thermal contact with the coldest exhaust air. There is still a temperature difference, so the cold heats up, and the hot cools down. Then if you think of the other side, the hottest exhaust air is exchanging heat with the hottest incoming air. Same deal, there is still a temperature difference, thus heat transfer.
But if you consider the numbers in the second case, the warm exhaust air could be cooled down to 16°C and still be exchanging heat with the 15°C air it is in contact with. And the incoming cool air can be heated to 24°C and still exchange heat with the 25°C hot exhaust air it is in contact with.
So instead of parallel flow to mix the temperatures, counter flow can maintain a constant small temperature difference to basically switch the two temperatures around. This also has the benefit of keeping the rate of heat transfer approximately equal at every point in the heat exchanger, so theres no "less efficient" spots.