Intuition/Motivation behind mixed states I have some confusion regarding pure, mixed and entangled states and I'm trying to gain some clarity on this.
Set up, and my current understanding:

*

*One fundamental distinction I seem to have (please correct me if I'm wrong) is that pure and mixed states are attributes when working with a single Hilbert space. However, to say that a state is entangled necessarily requires us to be able to decompose some Hilbert space $\mathcal{H}_{C} = \mathcal{H}_A \otimes \mathcal{H}_B$ and then check whether $|\psi\rangle \in \mathcal{H}_C$ can be written as some $|\phi_A\rangle\otimes|{\phi_B}\rangle$ where $|{\phi_{A,B}} \rangle\in \mathcal{H}_{A,B}$ respectively.

*A pure state is defined as one for which the density matrix $\rho$ satisfies $\rho^2 = \rho$ i.e. is a projection operator. In the Wikipedia definition for pure states, it is given that a pure state is one which can be represented by a single ray in Hilbert space. However, suppose I work with some Hilbert space $\mathcal{H}_A$ with basis $\{|a_i\rangle\}$, then the basis states themselves are rays in $\mathcal{H}_A$ but some superposition of the basis $|\phi_A\rangle = \sum_{i}\phi_{i}|{a_i}\rangle$ would also be a different ray right? So a pure state could either be one of the basis states, or could be some arbitrary superposition of the basis states.

*In the same way, a pure state may or may not be entangled. For example, if I consider two spin $\frac{1}{2}$ particles, then the state $|{\uparrow \downarrow}\rangle = |\uparrow\rangle\otimes |\downarrow\rangle \in \mathcal{H}_{AB}$ is a pure, unentangled state since it can be written as a product $|\phi_A\rangle\otimes|{\phi_B}\rangle$. However, the state $|\psi\rangle = \frac{1}{2}\left(|\uparrow \downarrow\rangle - |\downarrow \uparrow\rangle\right)$ is an entangled state in $\mathcal{H}_{AB}$ (since it cannot be written as a tensor product of states from the two separate Hilbert spaces) but it is also pure (since it is a unique superposition of the basis in $\mathcal{H}_{AB}$ and hence a ray. I suspect I can also write $\rho = |\psi\rangle\langle\psi|$ thus showing that the density matrix is a projection operator).

My question:
If this is true, then where do mixed states come from? What are the states in the Hilbert space which are neither the basis vectors, nor the superposition of basis vectors (since they are both pure from my understanding)? What does it mean to say that a mixed state  cannot be written as a state vector but can be described as a density operator directly?
 A: Mixed states allow you to describe classical uncertainty, in addition to quantum uncertainty. Quantum uncertainty, as descriped by superposition of states, is familiar from an introductory quantum mechanics course. Consider the state $|\psi\rangle=\frac{1}{\sqrt{2}}(|\uparrow\rangle+|\downarrow\rangle)$. While there is a quantum uncertainty in whether a measurement will yield $|\uparrow\rangle$ or $|\downarrow\rangle$, there is no ambiguity as to the quantum state of the system. Expressed as a density matrix, this is written
$$\rho_1=\begin{pmatrix}
1/2&1/2 \\ 
1/2&1/2
\end{pmatrix}.$$
What if instead I told you there is a 50% the quantum state of the system (not the outcome upon measurement) truly is $|\uparrow\rangle$ or $|\downarrow\rangle$. This would be described by the density matrix $$\rho_2=\begin{pmatrix}
1/2&0 \\ 
0&1/2
\end{pmatrix}.$$
The difference between $\rho_1$ and $\rho_2$ is the off-diagonal values. These are known as coherences, and indeed reflect a quantum coherence (superposition) between $|\uparrow\rangle$ and $|\downarrow\rangle$ in $\rho_1$. The diagonal elements are called populations and are the same in $\rho_1$ and $\rho_2$ because there is in some sense the same amount of $|\uparrow\rangle$ and $|\downarrow\rangle$ in both density matrices.
Indeed, a general density matrix can be expressed as
$$\rho=\sum_ip_i|\psi_i\rangle\langle\psi_i|,$$
where the $p_i$ is the classical probability of the state $|\psi_i\rangle$ appearing in some particular ensemble of states. This is a classical mixture of pure states; you can think of it as a bag of pure states, just like the bag of marbles you study in a stats class.
A first motivating example is the introduction of finite temperature into quantum mechanics. As you may know from classical statistical mechanics, ensembles of states are used to describe the classical uncertainty introduced by finite temperature, which encourages disorder (occupation of more states) in a system. In quantum statistical mechanics, much of the story remains the same. However, we now must study ensembles of quantum pure states. Ensembles are described by density matrices, and the canonical (thermal) ensemble is of the form
$$\rho_T=\frac{1}{Z}\sum_i e^{-\beta \hat H}|\psi_i\rangle\langle\psi_i|,$$
where $e^{-\beta \hat H}$ is the quantum version of the classical Boltzmann weight and $Z$ is the quantum partition function.
A: It is better, I'd suggest and given this, to not think of vectors in the Hilbert space as "states", but as a sort of "raw material" for building states. In fact, there are approaches to quantum theory which discard the Hilbert space altogether.
It is actually density operators, $\hat{\rho}$, that are the states - in all cases. It is just that when we have a density operator $\hat{\rho}$ whose image (or "range") is a ray in the Hilbert space, we can then conveniently represent it by a vector that points along that ray. Thus such $\hat{\rho}$ can be "considered as though they were" such vectors - but note, already, even there this is a bit of a fudge, because as you should know, if you have a vector $|\psi\rangle$, then $\alpha |\psi\rangle$ for some complex number $\alpha$ is already considered to be "the same state". This is the textbook's handwaving away at that stage (and fulfilling its want to assign some kind of immediate meaning to vectors) that we're actually talking a density operator with ray-shaped image. That is, already at this stage, we really have made the state the set
$$\{ \alpha |\psi\rangle\ |\ \alpha \in \mathbb{C} \}$$
and not just the vector $|\psi\rangle$. A nontrivial density operator, in part, corresponds to making this set even bigger, to a linear subspace of the Hilbert space, i.e. now some nontrivial linear combinations are similarly being considered "the same state".
As for what pure and mixed states mean, the way I'd suggest is the following: if the observable set is tomographically complete, then a pure state, in the density operator sense, represents the state of knowledge of an agent possessing a quantity of information about the system equal to the maximum possible to have. A mixed state represents an agent with submaximal information. As @Chris suggests, one way to arrange that can be to flip a coin and present you with a system prepared ahead of time in some pure state without telling you which one it is. But that is not necessarily the only way. Another, much simpler, one is just to bring in a totally naive agent that hasn't been told anything at all. Then the only thing they can say is $\hat{\rho} = \hat{I}$, the identity operator (I believe).
