Why does wind direction significantly affect sound propagation? I live about 2 miles from a race track.
Some days the noise from testing is extremely noticeable, Other days despite the exact same testing activity, there is little-to-no noise at all.
This is mostly effected by the direction of the wind.
How does the wind direction affect sound propagation? Given the speed of sound vs the speed of wind is generally only a few percent, why and how does it make such a dramatic difference?
 A: Since the mechanism is so well written on Reddit in here, I won't even try to rephrase it. Here it is:

Yes, wind has an effect on the speed of sound, and this effect has
  interesting ramifications for the propagation of sound outdoors.
An acoustic wave, as you know, is a mechanical wave traveling through
  a medium. The sound we deal with most often is carried through the
  medium of air, at a speed of around 343 m/s.
Wind is the bulk motion of air in a given direction.
When you combine these two ideas together, you get that sound is a
  wave moving through a moving medium. Unsurprisingly, that means that
  the velocity of acoustic wave is equal to the speed of the wave plus
  the speed of wind in that direction. IE, if the wind is moving at 20
  mph (8.9 m/s), then sound will travel downwind at 351.9 m/s, upwind at
  334.1 m/s, and crosswind at the regular 343 m/s. Note that it takes a significant windspeed to appreciably alter the sound speed in any
  given direction.
What's really interesting is how windspeed gradients alter the path
  that sound takes through the air. As a rule, sound waves bend towards
  regions of lower sound speed (an effect known as refraction that's a
  direct result of Snell's law). Couple this with the fact that
  windspeed tends to increase with greater distance from the ground, and
  you find that sound refracts downward when moving downwind and upward
  when it's moving upwind.
Sound tends to emanate from sources in roughly all directions. Some
  goes towards the listener, some goes away from the listener, and some
  shoots up into the sky. When you consider the effect of refraction,
  the question becomes "Does more sound go into the sky or to the
  source?" As it turns out, it will depend on where you stand with
  respect to the wind.
If I'm listening to someone far away talking, and there is wind, I
  probably want to stand downwind of them. Why? Because the sound that
  normally goes up into the atmosphere will instead refract downwards
  towards me, focusing on me. Conversely, if I was standing upwind, the
  sound would refract up and away from me, causing me to receive a lower
  effective level.

@Farcher suggested temperature gradient in the comments which in my opinion has quite small effect.
Thanks to Reddit user therationalpi.
