I can't get something strait... Lets use the usual example: A man is flying away from earth on a very fast spaceship. So, His time is now not moving in the same "speed" like earth time but slower. But why do we say that the man is moving faster than earth? why is it not that earth is moving faster than the man? I mean, it's not like there is a direction that if you move that way then you are faster and the other way means you moves slower.
So, how can we assume that's time is moving slower for the man and not on earth? Maybe the man is flying the "other" way and now he is slower than earth... If the man is not accelerating than it could appear to him that earth is moving fast and he is stationary... isn't it?
So, I can sum it as: In relation to what do we say that something is faster than some other thing?