We don't have to worry about falling into the black hole because we have far too much angular momentum from our motion about the galactic center. This can be made quantitative by considering the sign of the effective potential of our orbit. For reference, the sun is about 27000 light years from the galactic center, and its orbital speed is about 220 km/s. You'll find that the centrifugal term overwhelms the gravitational term by a factor of about $10^5$.
Also, on a galactic scale, the mass of the central black hole is tiny. It is a few million times as massive of the sun. On the other hand, the total mass of all the stars in the galaxy is about $10^5$ times larger, and the mass of all the dark matter is another factor of 10 or so larger than that.
Careers in astrophysics have been built on the question opposite of the one posed here - how does material ever shed its angular momentum to feed the black hole and allow it to grow to the size we observe? The answer is quite detailed and complex, and is still an active research topic.