If you were to start a system from rest with, say, a low-pressure region in the middle, the initial movement would indeed be from the outer area toward the middle. As the flow accelerated, the Coriolis force would begin to curve the trajectories toward the right (in the northern hemisphere). The flow would continue to evolve until it reached a stable state: one in which the sum of the forces felt by a fluid parcel is just the force needed to keep it in a repeating orbit. At that point, the flow would persist.
So, when the combined force due to pressure gradient (inward) and Coriolis (to the right, which for a circular trajectory is outward) provides the right amount of centripetal acceleration to keep the fluid parcels moving in circles around the low pressure center, that swirling motion is going to last for a while.
Of course, the atmosphere doesn't do this ideal experiment--the air is moving all the time and never starts up from a standstill. But nonetheless, flow around high- or low-pressure regions tends to approach a stable circulation, which is why it's very common to see flows like this.