Prerelativity physics, Special Relativity, and General Relativity formalisms summary In order this have a better understanding of "the big picture", a tried to do the following summary, but I can't really complete it for GR. This based on the introduction chapter of Wald's book.

*

*Prerelativity physics:

-- spacetime: $\mathbb{R}^4$
-- symmetry group: $O(3)\ltimes\mathbb{R}^{3,1}$
-- referential invariant quantities : $\Delta t$ between two events, $|\Delta\overrightarrow{x}|$ between two simultaneous events
-- formalism: vectors (transform under $O(3)$)
-- causal structure: two events $p,q$ satisfy one of the three mutually-exclusive options: (1) it is possible for a material body to go from $p$ to $q$ (we then say that $q$ is in the future of $p$), (2) it is possible for a material body to go from $q$ to $p$ (we then say that $p$ is in the future of $q$), (3) it's impossible for a body to be at $p$ and $q$ ($p$ and $q$ are simultaneous). The set of all events that are simultaneous to $p$ is $3$-dimensional.

*

*Special relativity:

-- spacetime: $\mathbb{R}^4$
-- symmetry group: $O(3,1)\ltimes\mathbb{R}^{3,1}=ISO(3,1)$
-- referential invariant quantities: $\Delta s^2 = -(\Delta t)^2+|\Delta\overrightarrow{x}|^2$
-- formalism: four-vectors (transform under $O(3,1)$)
-- causal structure: same as before except that the third option is now replaced by the three new options: (i) $q$ is on the boundary of the set of all events that are in the future of $p$, (ii) $q$ is on the boundary of the set of all events that are in the past of $p$, (iii) it's impossible for a body to be at $p$ and $q$ ($p$ and $q$ are separated by a spacelike interval). The sets of all events in the last option is $4$-dimensional.

*

*General relativity:

-- spacetime: any connected orientable Lorentzian smooth $4$-dimensional manifold $\mathcal{M}$
-- symmetry group: ??
-- referential invariant quantities : ??
-- formalism: tensors (transform under what ??)
-- causal structure : ??
How to complete this for GR? Is the symmetry groups Diff($\mathcal{M}$) or the isometry group of $\mathcal{M}$ (group associated to the Killing algebra)?
Is there a more rigorous way of presenting this? In terms of group representations for example.
 A: GR just doesn't fit into the scheme you're sketching in the way you imagine. It's actually way more fun than that.
The symmetry group of SR is just the local symmetry group of GR. In real-world operational terms, "local" just means that SR works to good precision at small enough scales. In terms of the mathematical formalism, you have a tangent space at each point.
Diffeomorphism invariance isn't really a symmetry like the ones you're trying to analogize it to. In particular, it doesn't produce a conservation law when you plug it in to Noether's theorem. Newtonian physics can be stated in a way that has diffeomorphism invariance. Diffeomorphism invariance is just the requirement that we should be able to state our theory in such a way that it doesn't matter what names we give to the points.
The causal structure of a spacetime in GR isn't necessarily the kind of bipartite or tripartite system you describe for Galilean relativity and SR. All kinds of other crazy stuff can happen, e.g., a point can be in its own future light cone. If you want physics to be able to make predictions, then you want your spacetime to be globally hyperbolic, but it's unclear whether reality really is this way -- this is the cosmic censorship conjecture.
GR doesn't actually require that spacetime be orientable, time-orientable (which is a different thing) or four-dimensional. The formalism works fine if these assumptions don't hold.
