Does it matter in which direction I travel in relativity theory? I'm not a physicist, but I'm still very interested in the relativity theory, especially in how the twin paradox is explained. Actually, it does not make sense to me and I hope you can answer my following question to help me understand how it works:
Does it matter in which direction i travel in the relativity theory?
When I first read about the theory, I thought, that it does not matter if A moves away from B, or B moves away from A, because every party will see the effect of "time going slower" for the other pary, meaning A sees B aging slower and B sees A aging slower.
But regarding the twin paradox, when A leaves the earth with insane speed, and returns years later, A has aged less than the people on earth. If this is actually true, then my previous assumption is wrong and it does matter if A moves away from B or the other way round, because the effect could be turned around, so that the earth ages less than A, if the earth moved away from A instead.
If this is true, then this brings up a next question: How can we know if B moves away from A or A moves away from B? Since the earth, our solar system, and even our galaxy is already traveling in the universe at some speed in a specific direction, we could actually stop moving by flying in the exact opposite direction at the same speed. but would this be considered as the earth moving away or as we moving away?
I really hope you can help me with that :)
Update for @Alfred Centauri's answer
the stay-at-home twin does not change direction while the travelling twin does.
When I look at it from only the travelers perspective, then yes. But when I look at it from the earth's perspective, then the earth indeed changes its direction, while the traveler does not. Why is that changing direction thing that important?
As I could see in Will's answer, the change of speed in universe seems to be the critical difference. In the time where the traveler is still on the earth, both the traveler and the earth move at the same speed in the universe, which means they age exactly the same. As soon as the traveler starts to accelerate or decelerate (move faster or slower than the earth) he slows his own time, no matter if he accelerates or decelerates.
If this is true, this means that not only the relative speed of the objects influences the time. Also the change of speed in the universe does.
But this opens another question: What if 2 travelers move to the exact same position in the universe, but traveler A goes there at twice the speed? Are the two travelers the same age when they meet at the destination? Or is one traveler older because he moved faster?
 A: 
If this is actually true, then my previous assumption is wrong and it
  does matter if A moves away from B or the other way round,

In the context of the Twin "Paradox", it doesn't matter.  It isn't the direction that makes the difference, it is the path through spacetime.
The key to understanding the twin paradox without the frightening mathematics of GR is this:  from any IRF (inertial reference frame), the stay-at-home twin does not change direction while the travelling twin does.

For example, chose as your reference frame, instead of the one above, the frame of the travelling twin on the outbound leg (we've switched which twin is "moving away").

In this frame, it is the stay-at-home twin that ages slowly, during the outbound phase, compared to the travelling twin.  However, and crucially, the travelling twin must change direction in order to return to the other twin.
So, at the halfway point, the travelling twin changes direction and is now travelling faster than the stay-at-home twin.
Now, it is the travelling twin that is ageing more slowly than the stay-at-home twin.  Moreover, due to the non-linear time dilation factor, the travelling twin is ageing slowly enough on the inbound leg such that the total ageing along the outbound and inbound paths is less than the total ageing along the straight path of the stay-at-home twin.
The bottom line is that the situation isn't symmetric.  The stay at home twin never changes direction while the travelling twin does.

But when I look at it from the earth's perspective, then the earth
  indeed changes its direction

No, from any inertial (non-accelerated) reference frame, the stay-at-home twin's path through spacetime is straight.
You're not taking into account that the travelling twin changes from one IRF (the outbound leg) to another (the inbound leg), i.e., the travelling twin boosts from one frame to another while the stay-at-home twin does not.  The travelling twin's frame of reference is non-inertial during the turnaround.
This is the crucial difference between the two paths through spacetime; one has a "kink" (the turnaround), and the other does not.

Why is that changing direction thing that important?

Because a straight spacetime path connecting two events is different from a curved or kinked spacetime path connecting the same two events.
Intuitively, in Euclidean geometry, the shortest path between two points is a straight line.
Counter-intuitively, in the Minkowski geometry of spacetime, the straight path between two events is the "longest" in the sense that the elapsed time (proper time) along a straight path is larger than for any other path.
Since the stay-at-home twin's path is straight, the ageing along that path is greater than for any other path.
A: With respect, OP, I think you've missed the point of Alfred Centauri's answer.  As has been referenced in our other question, there is an unambiguous notion of acceleration in special relativity, which all inertial observers will agree upon.
In the twin paradox problem, it is taken for granted that the Earth is inertial, that no one is firing a rocket to try to change its course through spacetime.  Even if you included gravitational effects, the Earth would still be inertial--the notion of acceleration in GR explicitly does not include gravity, as gravity merely changes the geometry of spacetime.  To see this, consider a sphere and a great circle on that sphere.  Two great circles intersect, but they are nevertheless the distance minimizing paths on a sphere, and for this reason, they are considered "inertial" or are called geodesics.  Geodesics in GR are inertial paths, but they instead maximize time between events.
Anyway, GR is not pertinent here, as the twin paradox manifests itself in SR.  The relevant point is the derivative of velocity with respect to proper time.  This is the notion of acceleration needed to understand the paradox.  In the twin paradox, the Earth has no such change of velocity, while the twin who leaves does as a result of firing a rocket to alter course and return to Earth after initially departing.
But Muphrid, you might say, the twin who left perceives the Earth as changing direction also.  His idea of the Earth's velocity is relative to him, and as he fires his rocket, he thinks the Earth is accelerating.
No, no, absolutely not.  All measurements are done with respect to a locally inertial frame.  Any simple accelerometer will tell the twin who left that he is the one accelerating.  All he has to do is hold out an apple and watch it fall in the cabin of his rocket.  The apple is inertial; the twin's body, with the structure of the ship acting upon him, is not.  He will perceive the Earth's velocity changing, but all these changes can be attributed to his own frame changing over time.
In other words, it would be like taking an arrow that points north and saying it's changing when, in fact, you're just spinning on a merry-go-round.
It is this acceleration of the rocket twin that leads to the difference in ages.  As we said, geodesic (or inertial) paths are the longest in terms of perceived (that is, proper) time.  Thus, the Earth has a longer proper time than the rocket does.
A: The key difference in the motions of observer A and B is that A needs to accelerate at some point to return to Earth, whereas B remains stationary with respect to the Earth's frame. Each observer can set up experiments to determine who, in fact is accelerating as they will experience a force. One can in fact study this problem with only special relativity as follows. We will calculate the proper time for each observer, calculated in the frame of observer B (Earth). Using that the infinitesimal change in proper time $d\tau$ for an observer moving at an instantaneous speed $v$ is
$d\tau = \sqrt{1-v^2}dt$
for B this means $d\tau_B = dt$ and for A, moving at some instantaneous speed $v(t)$, we have $d\tau_A = \sqrt{1-v(t)^2}dt$. We now see that independent of the actual path the accelerated observer follows:
$d\tau_B > d\tau_A~~~$ as $~~~v(t)>0$
and so, integrating over the entire $t$ range for a return trip for observer A we have
$\tau_B>\tau_A$
and so the paradox is resolved and observer B is older than observer A.
