Quantum states: wave functions or operators In a typical quantum mechanics course/textbook for physisists, and even in Hall's Quantum Theory for Mathematicians, states of a quantum mechanical system are said to be vectors in some Hilbert space $\cal H$. I have found some other seemingly more precise sources, for example Takhtajan's Quantum Mechanics for Mathematicians, that say that states are in fact trace class bounded operators on $\cal H$. Particular instances of the latter are 1-dimensional projections onto vectors in $\cal H$, so this second view encompasses the first.
I am unsure whether or not there is something wrong with the first model and this is the reason for introducing the second one, or is the second one simply a mathematically more challenging extension of the first one and this is the reason why textbooks deal only with the simpler one? Is this something similar as in classical mechanics, where you might first study only Newton's equations and only later introduce Hamilton's approach?
 A: The second view is strictly broader than the first, and it is used to describe mixed states as well as the pure states described by the first view.
There are several different situations that require the use of mixed states, but the simplest to understand is that of a (classically) probabilistic source for the states. In this paradigm, you have a preparation procedure which normally results in the pure state $|\psi⟩\in\mathcal H$, but, for whatever reason, occasionally bugs out and produces something else. Thus, with some (say) 5% probability, the preparation procedure will actually give you some other $|\psi'⟩\in \mathcal H$, and for whatever reason you cannot identify these cases before they are used or post-select them out afterwards.
The way to describe these situations is using the formalism of density matrices. Any and all quantum mechanical predictions for experimentally measurable quantities will essentially involve a matrix element of the form
$$
A=⟨\psi|\hat A|\psi⟩.
$$
The density matrix formalism works by a clever re-phrasing of this expectation value into the form
$$
A=\mathrm{Tr}\left(\hat A|\psi⟩⟨\psi|\right),
$$
which can easily be seen to give the same quantity (either by direct computation in a basis, or by choosing a basis with one member along $|\psi⟩$).
In our situation, however, we're contemplating a preparation procedure that might produce a bunch of different states, $|\psi_j⟩$, with probabilities $p_j$. Each of these will produce an expectation value $A_j=\mathrm{Tr}\left(\hat A|\psi_j⟩⟨\psi_j|\right)$ for our quantity, and we then need to average those results, weighed with the corresponding probabilities $p_j$. Because of the linearity of the trace, and the clever choice of our representation, we can encapsulate all of the preparation procedure into a single part of the expression:
$$
A=\sum_jA_j=\mathrm{Tr}\left(\hat A\sum_jp_j|\psi_j⟩⟨\psi_j|\right).
$$
This means that the true descriptor of the system is the density matrix $\rho=\sum_jp_j|\psi_j⟩⟨\psi_j|$, and it cleanly gives us the expectation value of any operator $\hat A$ via $A=\mathrm{Tr}(\hat A\rho)$.

Now, the existence of a completely classical probabilistic source of quantum states is not entirely without challenge, but either way you're stuck with density matrices: the alternative is to have a state that is entangled with its environment, and when you decide to ignore the environment (because it takes no further part in the measurement) what you're left with is also a mixed quantum state for the system.
Past a certain point, then, mixed states are a necessity, and whenever those are around you need to use the trace-class operator (the density matrix) as your descriptor of the state of the system.
A: States of quantum mechanical systems are in general functionals of the W* algebra of observables of the system, that are positivity preserving and of norm one.
All quantum states cannot be represented at the same time as vectors on a given Hilbert space. All the so-called normal states are representable either with the orthogonal projection on a single vector, or with a positive trace class operator with trace one. There exist however also non-normal states (at least for algebras of observables irreducibly represented on infinite dimensional Hilbert spaces). These non-normal states are, however, often considered "unphysical" (I think mostly because nobody ever observed a phenomenon that was explainable only with the system being in a non-normal state).
Let us restrict to normal states only. There is an important distinction between pure and non-pure states that is in some sense related to the distinction between vector-states and trace-class states. The pure states are the ones that carry maximal information on the system, while the mixed ones have an incomplete information, and may be thought as a "statistical mixture" of pure states. The pure states are the ones that yield an irreducible representation of the algebra of observables (via the GNS construction). In a given irreducible representation, all the pure states are given by rank one projections on vectors of the Hilbert space, while the mixed states are given by trace class operators that are not rank one.
