Summary
I believe you are absolutely correct, but, on the whole, this is how we want things to be - for this is exactly how the Equivalence Principle is encoded in the geometric General Theory of Relativity.
I'm not altogether sure I fully grasp this particular text[1], but it seems to me the author is simply saying that geometric axioms that allow transformation to inertial frames are only a small subclass of all possible axioms, and therefore GTR is perhaps not "impartial" or "democratic" as Einstein believed it to be. This too is probably true, but one has to choose an approach and the one chosen by GTR would seem to be a good „Occam's Razor“ compatible choice.
Details
It is true that once we postulate that spacetime is described by a semi-Riemannian (Lorentzian in relativity) manifold, then we always get the possibility of transformation to inertial co-ordinates as a diffeomorphism "for free" from the postulate. Indeed, Brown talks about this in the text in his discussion of Riemann Normal co-ordinates[2]: for any point $p$ that any co-ordinate system is diffeomorphic to one which is inertial at $p$, in the sense that uniform motion along the co-ordinate lines is truly inertial motion at $p$. It may not be (and, in curved space is not) possible to achieve this condition at all points in a neighborhood of the point, but it can be done for any one point and this is all we need to define inertial motion.
Indeed, this is how the equivalence principle is encoded into GTR: it is always possible for a small enough body to freefall and feel no forces - Galileo could always drop his balls for students with the standard result - (at least, this is a thoroughly reasonable physical postulate) so therefore we need to ensure that the geometric description always allows a transformation that makes the co-ordinates centered on such a body Minkowskian. And the assumption of a Lorentzian manifold guarantees this.
So, it seems to me, the considering of only classes of co-ordinates diffeomorphic to ones that annul forces on a body has a real World, experimental justification. We want things to be like this to square with our observations.
Brown is right that this would probably not be the only way whereby one could encode the basic notion of equivalence: he/she seems to take issue with the need for a metric: perhaps there might be a sensible definition of Equivalence in simply a differentiable or even simply a topological manifold. But, given rulers and clocks are always present in Einstein's thought experiments, the Lorentzian manifold choice seems to be the simplest one: an Occam's Razor kind of action given the mathematical tools and paradigms prevailing in Einstein's time. One has to begin somewhere.
Notes:
[1] These are the words of the mysterious Kevin S. Brown. It's not clear at all who he / she is, other than the author of some highly respectable content at mathpages.com. In all the content he/ she seems to author, I can only find him/her referring to the name / pseudonym „Kevin S. Brown“ once.
[2] For any point $p$ in a semi-Riemannian (Lorentzian in relativity), any atlas of charts contains at least one one that lays down co-ordinates for a neighborhood $\mathcal{N}_p\ni p$ of $p$, by definition. And as long as the manifold $M$ is semi-Riemannian, given any point $p$ and element $X\in T_p(M)$ of the tangent space $T_p(M)$ at $p$, a geodesic through the point with nonzero tangent $X$ there is uniquely defined. Indeed, for a small enough neighborhood, one can define Riemann Normal (also called Exponential or Geodesic) co-ordinates, which label every point in the neighborhood by a unique element of the tangent space: the vector $Y\in T_p(Y)$ defines a point by $\exp(1\cdot Y)\,p$, i.e we keep parallel transporting $Y$ along the geodesic it defines and we do so until the affine parameter along the geodesic clocks up 1 unit. Then we stop and the point we have arrived at is the point defined by $Y$ in the exponential co-ordinates.
We can also choose an orthonormal basis for the tangent space $T_p(M)$ and we can choose it so that one unit basis vector is timelike, the others spacelike.
