When we talk about speeds in relativity theory, where are they measured? I recently asked a question here about if the direction we travel matters in relativity theory:
Does it matter in which direction I travel in relativity theory?
After I got answers and making more research about the topic, I found out that I was talking about speeds relative to the earth all the time. So, when I for example said that I travel away from earth at 50% of the speed of light, I automatically assumed that this is also the speed I can use to calculate the time dilation.
But since the earth already travels at some always varying speed and thus is not standing still in space, I think I also need to put this into the calculation.
I mean, if I would travel in the exact opposite direction the earth currently "flies" and at exactly the same speed, this would mean that I'm standing still and that the time for me would actually go as fast as possible, while the time on earth is still slowed down, as it always is due to its speed.
So: What I wanted to show you is that it really seems to depend from where I actually measure the speed I travel and what I consider the "root".
Moving away from earth at some speed can mean moving faster in space OR moving slower in space and thus also mean slower or faster time flow than the one on the earth.
Am I right with this? Or am I totally wrong and it's valid to measure the speed I travel by how much the distance from me to the earth gets bigger?
UPDATE
Actually, I think I have understood the answer of @John Rennie, and it really seems to explain a lot of the issues I ran into before. 
His answer was, at least to my understanding, the following: If two objects in space stay together, they share the sime timeflow. If the distance between the objects changes over time, we cannot say which one will see the others clock going slower and which one will see the others clock going faster. To say that, we need to go back to the change of velocity that led to the increasing/decreasing distance. Here, we need to determine which of the objects changed it's velocity by the method John mentioned. And this will lead us to the answer.
This also explains the twin paradox perfectly. But after doing some thought experiments I came across a scenario that I cannot explain:
Imagine we humans have indeed built a death star as a moving base in space. This death star has also a lot of smaller fighter ships on board. Now the death star starts to fly away from earth at, say, 0.1c. According to the people on earth, the time on the death start now goes slightly slower than the time on the earth. After a while, a fighter on the death star begins to fly twice as fast (0.2c relative to the earth, 0.1c to the death star) in the same direction to explore the forthcoming space. According to the earthlings, the time on the fighter goes even slower than the time on the death star. The guys on the death star confirm that and notice that the time on the fighter seems to go slower compared to their time. After a while, the fighter found some enemy and decides to return to the death star to alert them.
Here, the fighter has 2 options:
1) since the death star is also on its way, he could just stop and wait until the deathstar comes across.
2) Stopping, and moving into the other direction to inform the death star faster and before he runs into the enemies.
The fighter goes for option 2) and flies back at 0.1c.
For the earthlings, the fighter is now getting closer at 0.1c, while the death star is still moving away at 0.1c. For the death star, the fighter is getting closer at 0.2c.
And that is the point where this starts getting weird:
For the earthlings, the fighters speed decreased from 0.2c to 0.1c, which means that the fighters clock is now going faster, while still being slower than that on the earth. Actually, since the death star and the fighter are moving at the same speed relative to the earth, they share the sime timeflow.
For the deathstar, the fighters speed increased from 0.1c to 0.2c, which means to them that the fighters clock is now going even slower than that on the death star.
Can it really be?
 A: 
Moving away from earth at some speed can mean moving faster in space
  OR moving slower in space and thus also mean slower or faster time
  flow than the one on the earth.

There is no speed relative to space, there is just relative motion between objects.

But since the earth already travels at some always varying speed and
  thus is not standing still in space,

The Earth is moving with respect to other objects such as the Sun, the galaxy, other galaxies, etc.
The more I read your question, the more I think you're of the belief that there is some absolute rest frame with respect to which all object's speeds are measured.
But there is no absolute rest frame.  In the context of Special Relativity, there is no place for a notion of "standing still in space".  From the article Absolute Rest:

We now ask the following question:  If we remove from a region of space and time every palpable thing (i.e., everything with which we
  can associate a rest frame), does there remain anything with which we
  can associate a rest frame? When the question is posed in this way,
  the answer seems to be clearly no, essentially by definition. (Only if
  there exist palpable but unmovable entities could the answer be yes.)

If you accept that there is a preferred inertial frame of reference, a frame relative to which all other moving inertial frames of reference are absolutely moving, then you necessarily reject Special Relativity.
A: In special relativity, everything is relative. This includes time dilation. As mentioned, it does not allow for a "privileged" reference frame because special relativity requires that all physics appears the same from every inertial frame.
You could travel at what you believe to be the exact opposite direction as Earth and at the same speed, but that would only be relative to a chosen stationary object, which is only stationary in that inertial frame. Then, when you calculate time dilation, you would be moving through time as fast as possible relative to your stationary object and Earth would appear to experience slower time passing.
The main point of special relativity, however, is that there is no way for us to tell which inertial frame is the correct one. How do you know that what you've chosen as the absolute, non-moving frame is the correct one? The answer is, you don't. Thus, when we measure velocities, we can only measure one velocity relative to another. If you travel away from Earth at 50% the speed of light, it doesn't matter what direction you travel. The time dilation will be the same because you are calculating your time dilation relative to Earth and vice versa.
The concept of an absolute frame (while nothing says it's impossible) is unknowable (at least from the perspective of special relativity). It would require the ability to communicate information instantaneously in order to determine the absolute rest frame and as far as I know, there is nothing to support this being possible. Thus, when considering things using special relativity, you must remember that everything is relative to something. Everyone is right and no one is.
A: I think this is an answer to Van's comment rather than the original question, but this is how you unambiguously measure acceleration in SR. Simply arrange a sphere of test particles around yourself then watch them. As long as the particles remain in a sphere centred on you know you aren't accelerating. If you are accelerating the sphere of particles will move relative to you, and from the rate of movement relative to you you can determine your acceleration.
Actually you don't need a sphere as just a single particle will do. The reason I used a sphere is that the same idea is used in GR, but here things get much more interesting. For example an astronaut orbiting the Earth is accelerating as viewed from the earth, but the astronaut feels no acceleration. However by watching his sphere of particles our astronaut can tell he's in a gravitational field because the sphere will change shape due to tidal effects. You need a sphere because obviously you can't get the shape change from just a single particle.
A: Relativity is confusing for me, too. The whole point of it is that there is no "right" frame of reference. I make it simple by thinking like this: The system that feels acceleration gets all the weird effects like time dilation. To travel away from earth at 50% c, you first need to accelerate - and then you're the one getting time dilated.
