Spacetime as a coset of a symmetry group

In the introduction to his nice PNAS paper on symmetry, David Gross said

Einstein’s great advance in 1905 was to put symmetry first, to regard the symmetry principle as the primary feature of nature that constrains the allowable dynamical laws. Thus the transformation properties of the electromagnetic field were not to be derived from Maxwell’s equations, as Lorentz did, but rather were consequences of relativistic invariance, and indeed largely dictate the form of Maxwell’s equations. This is a profound change of attitude. Lorentz must have felt that Einstein cheated. Einstein recognized the symmetry implicit in Maxwell’s equations and elevated it to a symmetry of space-time itself.

One possible way of saying the same thing, in my mind, would be along the lines of "by fixing the symmetry group for the equations of motion as $ISO(1,3)$, one is forced to replace the Euclidean geometry of spacetime, $\mathbb{R}^4$, with the Minkowskian geometry $\mathbb{R}^{1,3}$." Mathematically, I think this boils down to replacing the coset space for spacetime with Galilean symmetry

$$ISO(4)/SO(4)\cong\mathbb{R}^4$$

by the following coset space of for positions/velocities under (special) relativistic symmetry

$$ISO(1,3)/SO(1,3) \cong \mathbb{R}^{1,3}.$$

Along these lines, I am wondering:

Questions: What is the reasoning that might lead one to consider these specific coset spaces as giving "spacetime." In other words, is it possible, by simply regarding a certain symmetry group as the full symmetry group of spacetime, can one derive the relevant spacetime itself? Are there any (physical) principles that would lead one to look at cosets of symmetry transformations fixing the origin?

• Full disclosure: I'm not entirely sure that this question makes sense, as is, but I hope that any insight you might have will help clarify things a bit. – user99292 Dec 24 '15 at 3:23
• Even though this may sound trivial, physics doesn't go from mathematics to observation but from observation to mathematics. Einstein didn't start out with a mathematical principle. He started out with what he already knew about physical observations. His paper simply contains his conclusions about his empirical knowledge. A physicist doesn't think about spacetime as mathematical spaces but as a fundamental physical phenomenon. The mentioned math, as pretty as it may be, captures only the most trivial aspects of it and is misleading if you don't know the entire phenomenology. – CuriousOne Dec 24 '15 at 9:00