What defines the reference system to determine the angular speed of the Earth around the Sun? Imagine there is only the sun and the earth. We could chose our reference frame in such a way that the angular speed around the sun is zero. In a $xy$-plane the $x$-axis always points to the earth. If we'd choose our reference frame like this centrifugal and gravitational forces would no longer be in equilibrium alas it is wrong. But what does define how that reference system has to be aligned?
From our defined laws about equilibrium we know how it should be but what is it that defines that it is like that? Is it the mass distribution of the stellar system, if yes, which physical law defines that? 
 A: Well, we generally want our reference frame to be an inertial one. And one that follows the Earth rotating around the sun isn't one. So yeah, things look funky in that non-inertial frame. In the non-empty universe we use the stars as our reference, since motion with respect to them is so slow as to be negligible over a human lifetime. 
A: If you're inside a windowless spaceship, and you have little thrusters you can position to apply angular acceleration to the craft, you could keep changing your angular velocity a little at a time and seeing how various experiments come out. You could connect two masses to either end of a spring, and then see what the equilibrium length of the spring is in various orientations (masses at rest in the spaceship). You would find one orientation in which the equilibrium length was shortest. That is the axis of rotation. With your thrusters, you apply some torque about this axis.  If you find that the spring is more stretched afterward, then apply some torque in the opposite direction.  After a few iterations, you reach a state where the length of the spring is the same regardless of orientation.  Now you're in an inertial reference frame, and you found it without looking at the stars.
In short, one way to define an inertial reference frame is to say that it is a frame in which the laws of mechanics may be applied without fudging by adding in fictitious forces.
The question of whether the frame you find by the process just described depends in some way on the fixed stars or not is difficult to answer definitively, and, to my mind anyway, has more to do with how we choose to think about things than with determining the outcome of any experiment. We can't, for instance, do the experiment of starting the entire universe rotating to check whether frames fixed relative to the stars would still be inertial. If we somehow could, and we did, and we found that yes, such frames are still inertial, then how would we know we had indeed spun the universe?  How could we measure the rotation rate?
A: The choice of z-axis seems fairly obvious: it should be perpendicular to the plane of Earth's orbit (a.k.a the ecliptic plane). You may then think picking an x-axis is arbitrary, but it's not. Because the rotational axis of the Earth is tilted (by 23.4 degrees), there is a preferred choice called the vernal line. It is the line created by the ecliptic plane and Earth's own equatorial plane (recall your highschool geometry: a line is formed when two planes intersect). This vernal line aligns itself with the sun twice a year: once at the spring and fall equinoxes. It is natural to want to choose an x-axis that aligns with the vernal equinox because it makes coordinate transformations easier.

Most astronomers use this type of coordinate system aligned with the vernal line and the ecliptic plane. When you are not interested at where the Earth is at, such as a space mission to Mars or Jupiter, then a distant star is typically used to define an x-axis, and the z-axis is kept as the ecliptic plane.
