But doesn't that assume that matter interacts with ether to create this ether wind?
No. It is simply enough to note that there is no frame which is at rest relative to all reference frames. In Galilean relativity, a body's observed speed is in general changed by a change in the observer's motion state: a change is wrought unless the motion state change is defined by a very special velocity vector that adds to the observed velocity vector to bear the latter along an equal-speed circle. An there is no observer motion state change that will preserve the speeds of two or more general equal speed bodies.
Actually in most aether theories, it was pretty essential and obvious that aether did not interact with what physicists called "matter" in the nineteenth centuries - otherwise there would be all kinds of observable effects such as drag and force of otherwise unexplained origin on material bodies, and that's not what science - even in the nineteenth century - observed.
Ultimately, for all the discussion of the Michelson-Morely experiment and all the considerations that led to special relativity, the only property an aether needed to have was to merely define a frame of reference relative to which light would have the propagation speed foretold by the relevant dynamical equations, i.e. Maxwell's equations. Later, even this property was eviscerated when Lorentz formulated Lorentz Aether Theory which is identical in its experimental predictions to special relativity. User ACuriousMind summarizes this theory, and why special relativity is preferred, in his answer here. Also, some more information in my answer here.
Once Maxwell postulated his electromagnetic theory, there is a definite prediction of the speed $1/\sqrt{\mu_0\,\epsilon_0}$ of light and so one cannot easily avoid the question, "what frame is this predicted speed relative to?". One could not avoid the question in the nineteenth century because in Galilean relativity observed speeds generally changed by a change of the observer's motion state. And then, in the nineteenth century, one would look to other known wave phenomena - most notably acoustic waves - and witness that the speed of propagation predicted from the dynamic equations was always a speed relative to a certain frame, namely that which the acoustic medium was at rest in. So the natural inductive step was to assume that for light too there had to be some privileged reference frame at which light would move at speed $1/\sqrt{\mu_0\,\epsilon_0}$ relative to. Note that Maxwell's equations would accordingly change their form in frames other than this one, as is so for acoustic waves in frames other than the one at rest relative to the medium.