"Ground" is one of those ambiguous words with a few different meanings.
It's simplest to think about static electricity. We know the planet has a pretty balanced electrical charge, neither positive nor negative (otherwise the solar wind would neutralize it, or we'd all float off into the air). Connecting something to the planet is a reasonable way to prevent it building up a static charge so it can't zap you. (You can still build up a charge and zap it, however, which feels exactly the same)
In a home electrical system "grounding" is used a different way. Metal objects (like the outside of metal computers) are connected to the ground wire. If a live wire touches a metal object, it sparks and trips the circuit breaker. We do this because that's better than the alternative. But this mainly works because the ground wire is connected to neutral inside the electrical panel - it's not only connected to ground. We're really talking about neutralling, not grounding. (So why don't they just use the neutral wire? Because that would be really unsafe if the neutral wire would get broken accidentally. A lot of the electrical code is about what if stuff goes wrong.)
Okay, so if we just need neutralling, why does it have to be connected to the ground with this big metal pole (grounding electrode)? It makes sure the electrical system's ground is about zero volts compared to the earth ground. You don't want to stand on the ground, and touch a metal computer, and get a fatal zap. (The electrical system is big with lots of capacitance, and uses AC which can leak through capacitance. It could hurt you more than static.)
What about battery-powered circuits then? Do they have ground? Actually, yes, kinda. You see, when engineers say "ground" it's often just shorthand for "zero volts". And zero volts is whatever you say it is. Usually in a battery-powered circuit it's the negative end of the battery, so in this case "connect to ground" just means "connect to the negative end of the battery".
To your other question: if you touch the negative terminal and the dirt-and-rocks ground why don't the electrons just flow out of the battery and into the ground?
They do... but only a few. Then the battery is positively charged which keeps any more electrons from going out. To make it keep going, you need to return the electrons to the positive side. In fact, so few electrons go out that you'd hardly notice it even if the battery was quite high-voltage.