You can think of planets like wells or deep holes in the ground (gravity wells). Denser things fall to the bottom (rocks), less dense things rest on top of that (water), even less dense things on top of that (air), and finally the least dense thing on top of everything (vacuum).
The air, for the most part, isn't leaving the planet for the same reason the water isn't flying out of the ocean, gravity is holding it down.
It's important to remember that space doesn't suck. It's not that kind of vacuum. There is not a force pulling things into space in the same way there is not a force pulling air out of a hole in a balloon.