There are two separate issues here. One is that an affine space is similar to a global coordinate system but without a preferred origin, so this question is similar to "Why can't there be a single coordinate patch?" But also there is the question about what an affine structure does for you and what a metric does for you and why we need a metric and why an affine structure isn't giving us what we need.
Single coordinate patch?
I'll only briefly mention the single coordinate patch. If the global topology is equivalent to $\mathbb{R}^n$ then we don't need more coordinate patches, if it is equivalent to $\mathbb{S}^n$ then we only need two, but a well chosen patch can cover the manifold almost-everywhere in every sense of the word. And it would work well enough if you added enough special rules to describe correlations between curves heading out to infinity. I don't personally see how it helps, but you could make it work if you needed to.
What does an affine structure do for us?
We can put a metric on the single coordinate patch of the vector space (a bilinear form, a very restrictive kind of metric that is consistent with the global linear structure of the vector space), thus getting a metric on the set, without having to have multiple coordinate patches. This structure actually matches the theory of special relativity well, because it can respect different linear paths (linear inherited from the affine structure, i.e. all pairs of points in the path give vectors in the same 1-dimensional subspace) as inertial observers and treat them all equally, and not single out a preferred point or direction.
What does a metric structure do for us?
Single we can treat an affine space as a simple manifold with a particular kind of simple space, using a full manifold theory potentially allows us to do more. And multiple coordinate patches isn't the important part. The important part is to make a metric directly without restrictions that it support a global linear structure. Why? Because general relativity has a new physical insight that has to be incorporated somewhere. The insight is from the observation that all bodies are pulled along at the same rate. For instance a body out rest above the earth is pulled straight towards its center. So no matter their mass then all converge towards the center. This is similar in spirit to a bunch of people at the north pole that set off in different directions in groups leaving every 24 hours. Regardless of the mass of the particular person leaving that day, and regardless of the direction they head out, if they go in a straight-as-possible manner, the group will converge on the south pole.
Since we consider straight-line motion as a natural behavior not requiring further explanantion, then if the paths induced by gravity were explained as straight-lines, straight lines that converge, this can explain why all these different bodies follow the same paths. So all that is needed is to make a sense of straight line paths, and to make them converge. The affine structure can do the former (hence is good for special relativity), but not the latter. For the latter we want to set up a curved metric so the "straight-lines" can converge.
And now I'd like to point out that this curvature reproduces what we often call gravity, but is not what I'd call gravity. The curvature can exist far from gravitational sources and propagates and exists on its own, the curvature a bit later in a place is related to the curvature earlier and nearby and how the curvature varies. Just like electromagnetic fields a bit later in a place is related to the electromagnetic fields earlier and nearby and how the electromagnetic fields vary. (Technically, the curvature is second order and the electromagnetic fields are first order, so you need more variation, but still the same principle applies). Gravity, to me, is about how gravitational sources can change that expression about how the curvature a bit later in a place is related to the curvature earlier and nearby and how the curvature varies. If allows different curvature than otherwise would happen. Just as electromagnetic charges can allow different electromagnetic fields than otherwise would happen.
To summarize this last point, we could have curved spacetime and there would be curved paths in spacetime. And this could work fine to explain and model the empty spacetime between sources. But Einstein's equation is about how different sources can cause curvature to bear different relationships to itself than it otherwise would.
So, that's what a metric does for you
Why do we need a metric?
We want to say that different bodies move along the same paths in spacetime, and we want that motion to be natural, a metric can make those paths natural.
Why can't an affine structure do that?
An affine structure gives a linear type structure, so doesn't naturally make paths converge. We could put additional structure on top to do that, but one we have that, it bears little-to-no relationship to the linear-type structure and we'd never need the linear-type structure. So you can have it, but you'd not need it and end up with a different metric anyway to describe the converging paths. To be fair if you lived in deep space in a very flat region of spacetime far from large gravitational sources, maybe you'd never bother to develop General Relativity because gravity wouldn't be something you were used
dealing with. But if your world contains these natural converging paths, you want a theory that naturally makes these converging paths stand out as things that can be seen and predicted.
And that's why an affine structure isn't giving us what we need.
edit to respond to added part about straight lines
Straight lines with proportional tangents in an affine space never intersect. Einstein started with the physical observation that initially "parallel" straight lines can converge towards each other and intersect (for instance when there is a gravitational source between the paths they otherwise would have taken), so he sought a mathematical system that can encode that. Affine spaces are not that system because parallel lines do not intersect