Good question!

I think perhaps the situation you are describing would imply that we are in the center of the universe, i.e. that all of matter is expanding out from some small center, with some distribution in velocities.

This would be inconsistent with the dominant viewpoint in science, the Copernican principle, that says that our vantage point is not special. Indeed, of the the main assumptions in cosmology is that our universe is homogeneous and isotropic on large scales. I agree that this is not an argument. It is in fact circular reasoning. But it reflects the common view of scientists today.

(From a physics point of view, you could argue that gravity would cause the matter to slow down as it spreads out, but I'm fairly certain that this would reduce to the ordinary Friedman equations, so I don't think that would lead to any observable difference.)