The choice is exactly the same as for Euclidean space, which can be represented by the vector space $\mathbb R^3$, or by the associated affine space. Either works fine, and the only real difference is the existence, in the vector space, of a zero vector. In general, it is more formally correct to use the affine version, which has in-built formal support for translation invariance; however, this can be cumbersome and it is often cleaner to work with the vector-space version.
It's important to note that, even mathematically, affine and vector spaces are already pretty close to indistinguishable. More precisely, given an affine space $A$ with an associated vector space $V$ (i.e. such that a left action $l: V\times A\to A$ exists, with the appropriate identity, associativity and uniqueness properties),
- $V$ itself is also an affine space, since it acts on itself trivially, and
- $V$ can be recovered from $A$ by fixing some point $p$ to act as zero, so that each $v\in V$ is in one-to-one corresponding to the point $q\in A$ such that $q=p+v$.
It is therefore not that meaningful to "choose" between either definition, as the objects are so closely equivalent.
In practice, unless you explicitly want to deal with the translation invariance, you often take $\mathcal M$ to be a four-dimensional vector space. It is good physics practice, in any case, to distinguish between events and displacements, both of which can be elements of $\mathcal M$, but for which some actions - specifically, adding one event to another - don't make that much sense. If you do that distinction in practice - that is, if you follow common sense about what you should add to what - then you're essentially using the affine-space definition, without all the cumbersome formal machinery. It's simply a lot easier to remember that the origin, as an event, is not all that meaningful, and not go to the lengths of formally striking it off the books.