Time dilation due to change in speed I can't get something strait...
Lets use the usual example: A man is flying away from earth on a very fast spaceship.
So, His time is now not moving in the same "speed" like earth time but slower.
But why do we say that the man is moving faster than earth? why is it not that earth is moving faster than the man? I mean, it's not like there is a direction that if you move that way then you are faster and the other way means you moves slower.
So, how can we assume that's time is moving slower for the man and not on earth? 
Maybe the man is flying the "other" way and now he is slower than earth... 
If the man is not accelerating than it could appear to him that earth is moving fast and he is stationary... isn't it?
So, I can sum it as: In relation to what do we say that something is faster than some other thing?
Thanks!
 A: 
So, I can sum it as: In relation to what do we say that something is faster than some other thing?

We just arbitrarily decide that something is motionless. Very often we pretend that earth is the motionless thing, although earth revolves around the sun. 
We know for sure that earth is not motionless, so we can conclude that earth for sure experiences time dilation.
If an astronaut revolves around the sun faster than the earth revolves around the sun, then that astronaut experiences more time dilation than the earth.
When motions are revolving motions, or back and forth motions, we know which motion is faster. 
A little addition:
Earthlings looking at the sun must see:
1: time dilation as they see the sun moving
2: time ticking fast on the sun as they are revolving
Admittedly that is slightly odd. For an observer on a revolving platform things are slightly odd.
A: Special Relativity only applies to non-accelerating inertial reference frames, thus special relativity cannot be used to model your example.

But why do we say that the man is moving faster than earth? why is it not that earth is moving faster than the man?

That's a subtle misunderstanding of relativity - the critical factor is the frame of reference of the observer. If you are an observer on Earth (assuming Earth is not accelerating), you would observe the man moving with respect to you. If you were the man, not accelerating, you would observe the Earth moving with respect to you.
