The significance of the Poincarré group as a Symmetry group Any answer to my questions may assume arbitrary knowledge of differential geometry, which I will be happy to learn in order to understand the most appropriate type of formulation for the theory.
I would like to consider Special Relativity as the theory of a general Lorentz manifold $(M,g)$ where $M$ is diffeomorphic to (the standard diff. structure of) $\mathbb{R}^4$ and the metric (of constant Lorentzian signature $(-+++)$) is constant over all of $M$ in the coordinate-free sense. I do not, however, want to assume that $M=\mathbb{R}^4$ and that $g=(\text{usual matrix})$ because I find it unphysical to apply "linear transformations" to positions in spacetime. 
In particular, I demand that (for this discussion) the theory be formulated with arbitrary smooth coordinates (in the sense of differential geometry) where we can of course choose global coordinates, but no single coordinate system is in any sense "more fundamental" than another or "a priori innertial". This is as opposed to the case of $\mathbb{R}^4$, where one chooses as innertial the coordinate system $id$, which returns positions in spacetime as the tuples of numbers that they are, together with any other coordinate system which is connected to $id$ by a Poincarré transformation. 
I interpret physically the manifold as the set of all events in spacetime, and the metric as designating causality; in particular the metric dictates what (smooth) curves in $M$ are space-, time-, and lightlike. In order to specify past and future we assume the existence of a global vector field $X\in TM$, which is defined to be future-directed at each point. Together $X$ and $g$ specify the past and future light cones for each point $p\in M$ as subsets of $M$ (which is not considered as a vectorspace) as well as all(?!) other concepts relevant to causality, such as Cauchy surfaces and the like.
My questions:

Question 1. In the next step of this theoretical setup, which is not completely clear to me, one defines a linear action of the Lorentz group$^1$ (I don't think Poincarré makes sense here?!) on each tangent space $T_pM$. How does one set up this representation and justify physically why both the group and the chosen representation are meaningfull and the correct choice?

$ $

Question 2. Is it possible with the above prerequisites to meaningfully define when a diffeomorphism $x:M\to\mathbb{R}^4$ is an "innertial coordinate system"? Would this be equivalent to requiring that all "$g$-Levi-Civita-connection-straight" curves be mapped by $x$ to straight curves in $\mathbb{R}^4$?

$ $

Question 3. Assuming the above makes sense, what is the connection of the Poincarré group with coordinate changes between innertial coordinate systems?


Footnotes:
$^1$ I consider all these groups just as mathematical objects - Lie groups with a priori no physical significance. I want to include the physical significance by arguing why these groups with their appropriate actions on the appropriate objects are physically meaningfull.
 A: First off, your abstract setting is slightly off - it doesn't make sense to demand that $g$ is "constant in the coordinate-free sense" because $g$ takes values in $T^\ast_p M \otimes T^\ast_p M$ at each point, so you cannot compare the values directly to find out whether they are "constant". A more natural requirement for SR seems to be that $(M,g)$ be flat, non-compact and complete, which all are abstract, coordinate-free requirements making no reference to the groups you want to examine. Also an abstract, but somewhat "cheaty" way for your purpose appears to be to just require $(M,g)$ to be isometric to standard Minkowski space directly. Let me just remark that some might consider the *symmetries to be the more fundamental objects, and claim that requiring the Poincaré group to be the global isometry group is the "correct" abstract assumption. Now, for the appearance of the Lorentz and Poincaré groups in your setting:
On each tangent space, there is a natural action of $\mathrm{GL}(4)$ simply because it's a four-dimensional vector space. The Lorentz group $\mathrm{SO}(1,3)$ is simply the subgroup of $\mathrm{GL}(4)$ that leaves the pseudo-inner product on that tangent space invariant. There is no need to "define" its action, it comes naturally in this fashion. No representation is chosen, this is naturally the fundamental representation of $\mathrm{SO}(3,1)$.
An inertial coordinate system is simply an isometry of $(M,g)$ and $(\mathbb{R}^4,\eta)$ where $\eta$ is the standard Minkowski metric. Note that this is different from a general chart because charts are not required to be isometries, and indeed maps geodesics in $M$ to straight lines in $\mathbb{R}^4$
The notion of the Poincaré group now takes the idea of the Lorentz group as the group of "isometries" of the pseudo-inner product and makes it global: The Poincaré group is the group of isometries of $(M,g)$, or, equivalently, of $(\mathbb{R}^4,\eta)$. Here, an isometry of $M$ is a diffeomorphism $f : M\to M$ such that $f_\ast g = g$.
