Well, when you study special relativity in a "covariant way" (using the formalism of Lorentzian geometry and so on...) you realize that the structure of the spacetime isn't a mere vector space, but rather, that of affine space $[1]$. But, a small conceptual gap is annoying me.
General Relativity introduces topological manifolds as the most fundamental stage and then you elevate them into smooth ones. In fact, we can say (due to the very structure of a manifold) that they are a huge collection of $\mathbb{R}^{n}$ vector spaces.
Concerning the context of physics and the whole realization and physical meaning of a affine space: can I say that a general manifold is locally an affine space?
$$* * *$$
$[1]$ M.Kriele. Spacetime.