Are superposition quantum systems really in two pure states at once? I know that quantum systems can have properties of waves as well as those of particles.
Also, I know that the mathematical description of quantum states is probabilistic, i.e.
$$\Psi = \frac{1}{\sqrt{2}}|0\rangle + \frac{1}{\sqrt{2}}|1\rangle$$
is more or less adding the probability (here $1 / 2$) that when the quantum is measured its state is $|0\rangle$ and the probability that when measured the state is $|1\rangle$.
I've also understood that the act of measurement destroys the superposition. 
However, I'm wondering if before the measurement, i.e. during superposition, the quantum really "is in" those two different pure states ($|0\rangle$ and $|1\rangle$) - or maybe even in all states in between those two - at once, or if it is always only in one pure state (or one state between those two) and we only say, i.e. that it is having spin up and spin down at once because we are relating to the mathematical probabilistic description where both spins are involved, but in reality it can't have both states, not even while not being observed. 
In other words, if a die were quantum, could the die 1) show all numbers at the front side at once before measuring it and the measurement destroys this so that it only shows one of those sides then, or 2) is there always only one number at the front side, but we talk of superposition because we don't know, which one until measuring it?
 A: The answer depends somewhat on the interpretation of quantum mechanics you choose. However, if you take one of the standard ones, then a superposition of two states is not the same as being in both of these states simultaneously, nor as being in an unknown one of these states, nor as being in an unknown state "in between". Superposition is an entirely new idea that we don't have words in conversational English to express.
It might help to give a real-world example that mirrors the math. You can drive your car in any direction. Suppose your car also has a widget that indicates what compass direction you're moving in, but it's primitive and can only say "North", "East", "South", or "West". These are the only four things it can ever say. If you drive northeast, it has a 50/50 chance of saying North or East. Now I'll reframe all your questions in this context and the answers should be clearer:

However, I'm wondering if before the measurement, i.e. during superposition, the quantum really "is in" those two different pure states (|0⟩ and |1⟩) - or maybe even in all states in between those two

"I'm wondering if, when you drive northeast, the direction of your car really is in these two different pure directions (North and East) - or maybe even in all directions in between those two?"

or if it is always only in one "pure" state (or one state between those two) 

"Or is the car always going one "pure" direction, like North or East?"

could the dice 1) show all numbers at the front side at once before measuring it and the measurement destroys this so that it only shows one of those sides then or 2) is there always only one number at the front side, but we talk of superposition because we don't know, which one until measuring it?

"1) Could the widget say North and East at the same time before looking at it? 2) Is the direction of the car always North or East, but we don't know which one until measuring it?"
Hopefully the answers to these questions are clear. The car can drive any way it wants to; the widget just doesn't tell us everything about the state, just like measuring if a quantum bit is 0 or 1 doesn't tell us everything about its state. Driving steadily northeast is just a fundamentally different thing from driving north and driving east and is not at all a probabilistic mixture of them. This is the way standard quantum mechanics thinks of superpositions. 
A: Insofar as what is "really" going on, nobody knows for sure. However, the following is an account I have been developing for a while that is based on some of the most modern work in the field of quantum foundations and interpretations, such as and including the new "quantum reconstruction" paradigms which seek the derivation of quantum mechanics from elementary postulates in a manner similar to Einstein's derivation of special and general relativity.
The general thrust that it seems much off this work is circling around comes to that a quantum state vector
$$|\psi\rangle$$
is not an object that should directly be attributed to a physical system. Rather, what it is is a mathematical representation, or model, of information which is held by an agent, about the physical system in question. Typically this is taken as a human experimenter in a lab when it comes to scientific usage, but there is absolutely no reason it needs to be. The only things the agent needs to be able to do are to store information, to acquire information from the external system, update its internal store accordingly, and, of course, have some way of deciding which of these actions to perform. In this setup, the "wave function collapse" is just the update of the agent's information store with new information.
To get a better handle on how that works, consider the following simple scenario. Suppose that you and someone else you know are going to go out from your house to somewhere. The other someone (I'll call her Kionna) had been using it prior. There is a tea kettle in the home, and you both go out the door and get in the car. Just as you do so, you ask your friend Kionna if she left the kettle on.
We can, in this mathematical setup, model Kionna's knowledge of the kettle as follows. Let her knowledge be denoted $|\psi\rangle_\mathrm{Kionna}$. There are two vectors, $|\mbox{kettle on}\rangle$ and $|\mbox{kettle off}\rangle$, representing the state of the kettle. If she knows it is still on, we have
$$|\psi\rangle_\mathrm{Kionna} = |\mbox{kettle on}\rangle$$
and if she knows it is off,
$$|\psi\rangle_\mathrm{Kionna} = |\mbox{kettle off}\rangle$$
as you might think. But what if you ask her and she says "Well, I'm not entirely sure. I think I turned it off", and you ask "are you sure?" and she says "Well, I think maybe 75% sure". What do you make of this? Would you say that, perhaps, you would find this not as informative, perhaps not as much as you'd like, as if she said "yes, it's on" or "no, I turned it off" and "I'm sure"? I'd expect you'd find it not.
That, essentially, is the informational interpretation of probability, and in the formalism here, we'd describe her knowledge with
$$|\psi\rangle_\mathrm{Kionna} = \sqrt{0.75}\ |\mbox{kettle off}\rangle + \sqrt{0.25}\ |\mbox{kettle on}\rangle$$
This, of course, is a superposition - and this is its interpretation: it means that the information she has is less than if she knew for sure it was in either state alone. She cannot give you as much information. Nonetheless - there is a way to go find out: either you go, or you can ask her, to go back in and look. If she goes back in, then she will, of course, see, and then her knowledge should be considered as changed, i.e.
$$|\psi\rangle_\mathrm{Kionna} = |\mbox{kettle off}\rangle$$
and you then hear this new datum and are relieved.
Likewise, the same goes with a quantum system such as, in your question, a qubit, where now the states are $|0\rangle$ and $|1\rangle$, or $\left|\uparrow\right\rangle$ and $\left|\downarrow\right\rangle$, as in the case where the qubit is the direction of a vector component of spin angular momentum. When it is in a superposition, that means that we (or the agent, more generally, that we attribute this information to) has impoverished information thereabout. But then that begs the question: if this $|\psi\rangle$ is "just" information held by the agent, and not by the qubit directly, then what is all the hoopla about? Why is this different from any other case where one is simply missing information, as we just used to illustrate these concepts? 
Well, the way it seems is that in many experiments, when it comes to a real quantum system, not only is it that our "agent" - such as a person experimenting on it in the lab, or anything else - that doesn't have the information, but that the qubit itself actually behaves "in reality" like its information is likewise limited! This is not sensible with a single qubit, because there's no way to empirically distinguish whether a result you get by asking for its value was there to begin with or not, or whether it was created at, or perhaps sometime between when you first became curious and, the time you looked at it, but requires many and prepared in specially-correlated "entangled" states to recover statistical phenomena that show that it is impossible for it to have always been in only states $|0\rangle$ or $|1\rangle$ all along or, at least, if that information did pre-exist, it would require a violation of relativistic causality under the hood, which seems to cast doubt upon it. Moreover, it can be derived that an agent conducting a query as to the state of the object must invariably change it on pain of gaining no information, and thus, we cannot be sure exactly how much information was there before we did so. That doesn't mean there wasn't any, but we can't know except perhaps for bounding it.
So the answer to "what happens when a system is put in a superposition?" or "is it in two states at once?" is that it's rather best thought of as being in a state in which it kind of doesn't know which one it's in. It's in a state where that there is only a fraction of a bit, if you will, specifying which of those states in, like a "splinter" of its classical, fully-resolved state. We cannot really visualize that, because our minds can are and only built for visualizing things that, effectively, can be expressed as a grid of pixels (even though a human brain doesn't literally work on that, hence why I said expressed), with classically many bits in each pixel. It is, in effect, the same situation as to trying to visualize 4 or more dimensions: it would require a higher-dimensional retina, and thus a higher-dimensional visual cortex, and thus a higher-dimensional brain altogether, to actually do it justice. The mathematical structures involved simply do not match or embed faithfully into the perceptual structures our brains operate on.
As to how exactly these things manage to exist in such a way that these parameters have these reduced, non-classical levels of information - now that's the part we don't know, and most likely, can't. This is actually a generic feature of empirical science: it only provides us with models. Where quantum theory takes this a step further is that it tells us that not only is this the case, but unless the theory is wrong, we cannot have any empirical means to even construct a model of the world that is not relativized to the knowledge of different agents therein and separable from their actions upon it: all such models are neither verifiable nor refutable. Attempts to construct such are basically what various other "interpretational" schemes like "many worlds" and so forth amount to, and thus there is no way to tell if any of them are "correct".
That said, what I will say is that one thing you may have heard in "popular" expositions of quantum theory, and a favorite jumping-off point for many "kooky" theorizers, which is that quantum mechanics says that a "conscious observer" is required or that "consciousness" changes reality by some "magical" way (like "psychic" powers) upon an observation, or whatever, or that "reality" is "all in our minds", is not required and not implied by quantum mechanics, any more than it is by classical mechanics. The reason some come to this conclusion is the need to include the agent in the description, but where it goes wrong is that it assumes additional qualifiers on the agents that are not at all necessary, such as that they be specifically conscious beings, when all that is "really" required is information processing. Of course, since conscious beings like ourselves are the chief users of the theory, they are also the chief agents we are concerned with, but that's a far cry from saying that the theory somehow demands or stipulates the need for specifically consciousness on the part of the agent we have to posit in order to use it as a descriptive tool. Nor does the theory require a belief that no "reality" which can be described independently of the agents exists: it constrains it, and it prevents building a verifiable model of it, but those are not the same as a statement of nonexistence. Again, you can believe that if you want, but it is a stronger position than is required.
A: 
However, I'm wondering if before the measurement, i.e. during superposition, the quantum really "is in" those two different pure states ($|0\rangle$ and $|1\rangle$) - or maybe even in all states in between those two - at once, or if it is always only in one pure state (or one state between those two) and we only say, i.e. that it is having spin up and spin down at once because we are relating to the mathematical probabilistic description where both spins are involved, but in reality it can't have both states, not even while not being observed.

A qubit in the state $\lvert+\rangle\equiv1/\sqrt2(\lvert0\rangle+\lvert1\rangle)$ is not in either $\lvert0\rangle$ or $\lvert1\rangle$. Saying that this represents a state that is both in $\lvert0\rangle$ and $\lvert1\rangle$ is closer to the truth, but still not accurate.
A superposition is fundamentally different than a statistical mixture of the corresponding physical realisations, as explained for example in this or in this other answers.
It is also tricky to say that a superposition is the same as the system "being in two different states at the same time", first and foremost because it is not clear what this even means. What is true is that a superposition is a physical state whose evolution will depend on both its amplitude of being $\lvert0\rangle$ and its amplitude of being $\lvert1\rangle$. In other words, the system is in both $\lvert0\rangle$ and $\lvert1\rangle$ in the sense that both of them contribute to how it will evolve in the future.
On the other hand, this phrasing is what leads to the most widespread misconception about quantum computing: that it works by simply exploring all possible computational paths at the same time. This is very wrong, and quoting the first thing you read in Scott Aaronson's blog:

If you take just one piece of information from this blog:
  Quantum computers would not solve hard search problems
  instantaneously by simply trying all the possible solutions at once.

This is why one should be careful in saying that "quantum systems are in different states at once": it makes one naturally think that things like this are possible, while they are not.

is there always only one number at the front side, but we talk of superposition because we don't know, which one until measuring it?

It is crucial to remember that this is not the case in quantum mechanics. Local hidden variables cannot explain some physical observations. The possibility of a local hidden variable theory (which you might understand as a "classical theory") explaining quantum mechanics can be experimentally tested, by observing the violation of the so-called Bell's inequalities (or some variety of these).
