In the case of liquids and gases, at least, there's no fundamental difference. To see this, take a look at Wikipedia's phase diagram for water.

Ignore the dotted lines for the moment, and note that the line between vapour (steam) and liquid stops at a certain point, called the critical point. What this means is that if you go through the following sequence of steps, you can turn liquid water into steam without ever noticing a sudden change:
increase the pressure above the critical point (about 218 atmospheres), keeping the temperature constant at a relatively high value, but less than $100^\circ C$
keeping the pressure constant at $>218$ atm, increase the temperature to above about $374^\circ C$
reduce the pressure back down to 1 atmosphere or below
reduce the temperature to somewhere closer to $100^\circ C$.
By doing this, you go from the region marked "liquid phase" to the region marked "gaseous phase" (or "vapour"), but you go around the critical point, thus avoiding going through the phase transition. So you never see the liquid boil, it just gradually changes its density until it becomes what we normall call a gas.
Now, you'll notice that as you do this you actually pass several of those dotted lines, so that you start with a liquid, then get a compressible liquid, then a supercritical fluid, then gas, then vapour. But these are, as you say, "human distinctions" - there's no directly observable change in the fluid when you pass the dotted lines, they just represent definitions that people sometimes find it convenient to make.
So for the liquid-gas transition, it's reasonable to say that although the phase transition (boiling or condensing) is real, measurable thing, the distinction between liquids and gases is not.
However, for liquids and solids there tends not to be a critical point - there's no path from one to the other that doesn't go through the phase transition. So in this case there's more motivation to say that they really are distinct. Though having said that, if you consider mixtures of several compounds it can get a bit less clear-cut. Some substances (perhaps most) can be a solid on small spatial scales but a fluid on larger ones - the Earth's mantle being a good example.
So basically I think you're right. In some cases the distinctions are real, and they're very often useful distinctions to make, but at a really fundamental physical level it's much more meaningful to talk about phase transitions than the reality of states of matter in themselves.