With respect to what does a Gyroscope maintain its orientation? Assume ideal conditions, no friction, no energy loss in any form.
Just an ideal gyro, in ideal conditions, spun into rotation and left alone. And placed somewhere on earth. 
From what I've read so far, it should maintain its orientation, with gimbals rotating as to maintain the orientation of the disc/plate.
But maintain its orientation with respect to what? That's the part that I can't absorb. I've seen an answer that says, "with respect to fixed stars"
I just don't understand how is that relevant at all. Even if we can't see their movement easily, doesn't mean that it doesn't exist. So I don't understand how is that a good reference point. 
It doesn't have to be a gyro. It can be a Foucault pendulum on north pole. 
 A: A gyroscope maintains its orientation with respect to any inertial reference frame. An inertial reference frame is one in which objects with no force on them remain at rest or in uniform motion (i.e., moving in a straight line with constant velocity).
A: Well, the reference of the gyroscope is inertia
As we know, inertia is that a force is required to change the velocity of an object.
In order to throw a weight you have to exert a force on it, changing it's velocity. 
The thing about inertia is that it is the same everywhere. At every point in space the same force results in the same amount of change of velocity.
Just try to imagine how the world would feel if inertia would would be randomly different from moment to moment, and from place to place. Then motion would be totally unpredictable.
But motion is very predictable, and from that we infer that inertia is the same, everywhere and in every direction.
This uniformity allows inertia to be a completely effective reference of motion.
what you can and cannot measure
As we know, you cannot measure what your position in space is, because space is featureless.
You cannot measure what your velocity with respect to space is.
But what you can do is continuously measure what your acceleration is, and you can continuously measure whether your orientation in space is changing or not.
Velocity is the time derivative of position, acceleration is the time derivative of velocity. As we know: taking a derivative is a reversable operation (up to a constant) 
With your acceleration and your orientation measured continuously you can retrace your path, kind of like walking back in you own footsteps. The fact that continuous logging of your acceleration will allow you to retrace your path demonstrates that inertia is an effective reference of motion
The Universe
We assume that everywhere in the Universe inertia is the same
It seems reasonable to assume that a spinning gyroscope remains in the same orientation with respect to the distant stars because in all of the Universe inertia is uniform.

The equivalence class of inertial ccordinate systems
The effectiveness of inertia to provide a universal reference for motion is referred to as 'the equivalence class of inertial coordinate systems'.
If you have a space and you want to assign a coordinate system to that space then the zero point of the coordinate system can be assigned to any point in that space. All of those coordinate systems are equivalent.
You can do a transformation that transforms a coordinate system to one that is oriented at a different angle. All of those coordinate systems are equivalent.
You can do a transformation that transforms a coordinate system to one that is moving with a uniform velocity to it. All of those coordinate systems are equivalent.
The members of that equivalence class have in common that there is no measurement that can single out one among that class as different from the others.
As we know: this equivalence does not extend to acceleration. The physical properties of inertia are the reference for the distinction between the members of the equivalence class of inertial coordinate sytems, and non-inertial coordinate systems.
