Good question!
I think perhaps the situation you are describing would imply that we are in the center of the universe, i.e. that all of matter is expanding out from some small center, with some distribution in velocities.
This would be inconsistent with the dominant viewpoint in science, the Copernican principle, that says that our vantage point is not special. Indeed, of the the main assumptions in cosmology is that our universe is homogeneous and isotropic on large scales. I agree that this is not an argument. It is in fact circular reasoning. But it reflects the common view of scientists today.
(From a physics point of view, you could argue that gravity would cause the matter to slow down as it spreads out, but I'm fairly certain that this would reduce to the ordinary Friedman equations, so I don't think that would lead to any observable difference.)
Edit: All this is considering only the evidence from redshifts. However, there is other evidence for the standard cosmology. These are nucleosynthesis (BBN) and the microwave background (CMB). I think that BBN would be unchanged, for the same reason as above, that the expanding matter just follows the same Friedman equations. On the other hand, the microwave background is radiation reaching us from some "last scattering surface" about 14 billion light years away (according to standard cosmology) in all directions. If we were all the matter in the universe, we wouldn't be seeing anything coming from there. There is also evidence that most of the energy density comes from a cosmological constant, which cannot be described by matter, only vacuum energy that really does cause expansion.