Why don't we assume a vector space structure for spacetime? At the outset I'll state that I understand completely why, physically speaking, there's no preferred frame - that's not what I'm asking in the question.
I'm not sure why we don't give spacetime a vector space structure though. I realize that doing so forces us to identify a certain unique point in spacetime as the "universal" origin. But in the mathematical formulation, I don't see what's stopping us from doing that. Even if we can't physically pinpoint a preferred point in spacetime, it's an entirely different matter to choose an origin for mathematical convenience.
What can go wrong with defining a vector space structure on spacetime even if we assume flat spacetime?
And in the case of general relativity, what can go wrong with assuming that curved spacetime is embedded in an $\mathbb{R}^4$ vector space?
 A: If you assume flat spacetime and cartesian coordinates, then there's no problem with defining an affine structure on spacetime, and indeed this is often done by various authors who call $x^\mu$ the coordinates of a spacetime displacement vector.  The situation is a bit more complicated in curvilinear coordinates, because the same vector will have different components depending on the base point to which it is attached, but this is not necessarily a deal-breaker.  At the very least, one could define the affine structure with respect to an underlying cartesian coordinate system.
A crucial aspect of a (real) affine structure is that starting from a given point $a$, a vector $\vec v$ defines an entire affine subspace, which consists of all points of the form $a + \lambda \vec v$ for $\lambda \in \mathbb R$.  Denote this subspace $\mathcal S(\vec v,a)$.

If a point $b\neq a$ lies in $S(\vec v,a)$ and $S(\vec u,a)$, then
$$b = a + \lambda_1 \vec v = a + \lambda _2 \vec u$$
$$\implies \vec u = \frac{\lambda_1}{\lambda_2} \vec v$$
and therefore $S(\vec v,a) = S(\vec u,a)$.  In this case, $\vec v$ and $\vec u$ are called parallel; we also say that $\vec v$ is tangential to $S(\vec v,a)$ at $a$.  If $a$ and $b$ belong to the same affine subspace, then a tangent vector to that subspace at $a$ must be parallel to a tangent vector at $b$ (this follows trivially by definition); we might therefore say that an affine subspace is an autoparallel, which means simply that all of its tangent vectors are parallel to one another.

In a space with nonzero curvature, this breaks down. Consider the space $S^2$ - the surface of a sphere.  Let our starting point $a$ be the north pole, and consider two vectors - $\vec v$, which takes us to Manhattan, and $\vec u$, which takes us to Paris.  These vectors would correspond to affine subspaces $S(\vec v,a)$ and $S(\vec u,a)$.  Note that straight lines do not exist on the surface of a sphere; however, the notion of autoparallel curves still survives.

The problem is that the south pole is in both $S(\vec v,a)$ and $S(\vec u,a)$, and via our analysis above, that implies that $S(\vec v,a)=S(\vec u,a)$, which clearly cannot be true.
The reason for the breakdown is as follows.  Given two points $a$ and $b$ in an affine space, there must be a unique displacement vector $\vec v = b-a$ which takes you from $a$ to $b$, and therefore a unique affine subspace of $a$ in which $b$ lies.  On the surface of a sphere, this cannot possibly hold, because any two autoparallel curves which pass through a given point will also intersect at the antipodal point.  As a result, there is no unique vector which takes you from a point to its antipode, and so the axioms of the affine space cannot be satisfied.

One could object to this reasoning on the grounds that $S^n$ is closed, and therefore cannot be thought of as a manifold built on $\mathbb R^4$ as a carrier set as specified in the question.  However, this can be countered with the following points.

*

*There is no particular reason that we should rule such spaces out. A closed FLRW universe, for example, would have $\mathbb R \times S^3$ as a carrier set, and in principle there's no reason to think that we don't live in such a universe.

*It is a generic feature of spaces with nonzero curvature that
autoparallels (which are also geodesics in the standard formulation
of GR) can intersect at multiple points.  Therefore, while the situation is    not necessarily as degenerate as the case of $S^2$ (in which every autoparallel passing through a point will also pass through its antipode), it remains the general case that there are points $a\neq b$ between which there is no unique geodesic, and therefore no unique way to assign a displacement vector to $b-a$.

*Even in the absence of such intersections, vector addition on an affine space must commute, so $a+(\vec v_1+\vec v_2) = (a+\vec v_1)+\vec v_2 = (a+\vec v_2)+\vec v_1$. However, in the general case, this does not hold; following $\vec v_1$ and then $\vec v_2$ along their corresponding geodesics leads us to a different point than following $\vec v_2$ and $\vec v_1$, as per the following example$^\dagger$.


It is precisely this non-commutivity which defines the presence of intrinsic curvature, and so we are led to the conclusion that if a space is curved, an affine structure (which necessarily includes commutative vector addition) cannot be defined.

$^\dagger$The pictured manifold is the surface
$$\mathcal M := \big\{(t,x,y)\in \mathbb R^3 \ | \ -t^2+x^2+y^2=1\big\}$$
In the $(u,v)$-coordinate system, in which
$$t = \sinh(u)$$
$$x = \cosh(u)\cos(v)$$
$$y = \cosh(u)\sin(v)$$
the metric (which is inherited via the embedding of $\mathcal M$ in (1+2)-dimensional Minkowski space)
takes the form
$$g_{\mu \nu} = \pmatrix{1 & 0 \\ 0 &\sinh(u)}$$
and the non-vanishing connection coefficients are
$$\Gamma^{u}_{vv} = -\cosh(u)/2$$
$$\Gamma^{v}_{uv}=\Gamma^{v}_{vu} = \coth(u)/2$$
A: 
What can go wrong with defining a vector space structure on spacetime even if we assume flat spacetime?

Actually, Minkowski space-time can be considered a $4$-dimensional vector space equipped with an indefinite,  nondegenerate symmetric bilinear form. Possible problem with the "universal" origin can be easily solved by introducing an affine space structure.
About a possible embedding in $\mathbb{R}^4$, things are not as simple. You may find some key information here.
