Are states of a quantum system represented by vectors in a Hilbert space? Is it true that the states of a quantum system are represented by vectors in a Hilbert space? 
When does the term "ray" come into play in this context?
 A: That's a rather delicate topic. I suggest you to begin with Section 2.1 ''Quantum Mechanics'' of Weinberg's ''The Quantum Theory of Fields'', Volume I.
All (normalized) wave vectors in the Hilbert space which only differ by phase  represent the same physical state of the system. All such wave vectors (corresponding to the same state) can be united into a ray in order to have a one-to-one correspondence between rays and pure states of the system. (There also exist mixed states of the quantum system - those which have to be discussed in terms of a more general formalism of the density matrix...)
So, here's the plan:


*

*You start with some vector space which you call the Hilbert space (with a positively defined scalar product).

*You limit yourself with the subset of normalized wave vectors (you only use those when describing the physical states).

*It turns out that those of them which only differ by phase represent the same state of the system.
$$
\begin{alignedat}{4}
&|\psi\rangle \quad&&\to\quad &&\text{some state of the system}\\
\operatorname{e}^{i\,\alpha}&|\psi\rangle \quad&&\to\quad&&\text{same state of the system}
\end{alignedat}
$$
where $\alpha$ is a real number.


Be mindful that even though the wave vectors representing the states should be normalized, during the calculations we often deal with the non-normalized ones. So, typically it's like that: do some calculations, normalize the result at the very end. However, one should remember that the operations of normalization and, say, addition do not commute.
A: Quantum Mechanical operations are all done in Hilbert Space. A Hilbert space is an infinite dimensional complex vector space. There are many reasons why we do so. For example, if you try to find the mechanics of  body by using the Hamiltonian formulation in Classical Mechanics, you will study the object's flow in phase space.  If you use use Lagrangian formulation the body's dynamics is seen in configuration space. But you should see that the system is unaffected by the formulation. The formulations differ only in their mathematical framework. Like wise in Quantum mechanics, you should realize that no observations are merely possible. I will explain why.
For a macroscopic body, you could observe or evaluate the body's properties like energy, momentum, position etc. without affecting the body's dynamics. You could make your measurement and tell the answer to any degree of accuracy you need. But when it comes to the microscopic particles, the observations or any try of making a measurement will change the system's properties. It is very fundamental property of quantum mechanical systems. Experiment like sequential Stern-Gerlach experiment tells us that. (You may also refer to the Quantum Double Slit Experiment, which is essentially a thought experiment.)
So this means you cannot measure any properties of a system at the microscopic level. Suppose you need to measure a particle's energy at this instant. If you make your observation, or experiment , that observation disturbs, more precisely alters the state of the system. So what you get is the result of your measurement, not the information that you wanted about the system , which is what we don't want.
Due to this, the story of quantum mechanics is formulated in  pure theoretical framework with precise postulates. Any classical properties correspond to an observable in Quantum mechanics. An observable is an operator that corresponds to a physical quantity, such as energy, spin, or position, that can be measured; think of a measuring device with a pointer from which you can read off a real number which is the outcome of the measurement. So instead of taking direct measurements, we use operators in quantum mechanics. Also the operators should act on something. The energy operator, for example should give the energy of the system. So the function on which our operator need to be operated should be such that it contains information about what we seek. So we define some possible states corresponding to a system which it could occupy, by the assumption that the system can occupy any one state at a single time. Our mathematical function can now be expressed as a linear combination of these states. That's how we define a state ket (a mathematical abstract for what we call the wave function). Now as in the case of Lagrangian and Hamiltonian mechanics, we define a space made of axes with each dimension corresponding to these individual states. It's like representing a vector in Cartesian co-ordinates by three base vectors. You could write a vector as a linear combination of these base vectors (x^, y^, z^).
But there is an essential condition that to be met inorder to write a state ket like this. The base kets you select should be eigen kets of the observable you are choosing. The dimensionality depends on the system and upon what state you are corresponding to. For example, in the Stern-Gerlach experiment, the experiment measures the spin of the electron in the incoming beam of atoms. There by the system (the atoms) have two degrees of freedom- one corresponding to spin up and the other spin down. So the system in terms of spin can be described by two base kets (we get a two dimensional vector space).
However there are situations in QM where there is a need for infinite dimensions. So, say the QM lives in an infinite dimensional vector space called Hilbert Space.
Now let's deal with your question. You should be confused by the statement "we are dealing with rays in QM rather than vectors". It means a state ket |x> and a constant multiplied by the same, say, a|x> where a is a complex constant both corresponds to the same state. For example, if you think about force and acceleration, the vector force is nothing but a scalar (mass) times a vector (acceleration). A scalar times a vector gives you a new vector with magnitude different, but both vectors sharing a common direction. So in terms of directions, acceleration corresponds to force. It is true. Whenever we say that a body is accelerating we know there is a corresponding force acting on the system. Otherwise, the force and acceleration correspond to the same state of the system.
That's why we say we are concerned with rays rather than vectors.
This has something to do with base kets while defining the state vector. I have mentioned that the state ket can be expanded as a linear combination of base kets which are eigen kets of an observable. While operating a state ket, say |x>, by an operator A, in general the new ket may not be proportional to the initial ket |x>.
But for an operator there are some kets say, |a'>, |a''>, |a'''. etc. so that the operator acing on the ket reproduces the same ket by a multiplicative constant. In such a case the kets |a'>, |a''>, |a'''. etc. are called eigen kets of the observable A.
i.e.,  A|a'>=a'|a'>;  A|a''>=a''|a''>; etc.
where a', a'' are the corresponding eigen values of the eigen kets.
In QM we are concerned only about such kets as they reproduce the initial ket. Our theoretical framework was such that the state ket should be unaltered by our observation or measurement of a property. While dealing with eigen kets of an operator, we could find the corresponding value of the observable without changing the state of the ket. This is the same as what we have said "in Qm we ware concerned only about rays rather than vectors".
Due to this reason, we will expand our state ket in terms of the eigen kets (the base kets of Hilbert space where we define our state ket).
See for reference: Modern Quantum Mechanics by J.J.Sakurai
A: States of a quantum system come in two different types, pure states and mixed states. Pure states are the states where you have maximum information about the system and mixed states are states with additional ignorance on part of the observer or the apparatus.
Pure states of a quantum system are represented by a special class of self adjoint operators $\rho: \mathcal{H}\rightarrow \mathcal{H}$ that satisfy the following.
i) Unit trace,  $\Sigma_{n=0}^{\infty} \langle e_n,\rho (e_n) \rangle =1$
ii) Non-negative, $\forall \psi \in \mathcal{H}, \rho (\psi ) \geq 0$
Now, any pure state $\rho: \mathcal{H}\rightarrow \mathcal{H}$ can be represented using a $\psi \in \mathcal{H}$ in the following way
$\forall \phi \in \mathcal{H}, \rho(\phi )= \frac{\langle \psi, \phi \rangle}{\langle \psi, \psi \rangle}\psi$
Which is why we often write $\rho_{\psi }$ instead of just $\rho$.
This is where the notion of a state being represented by an element of $\mathcal{H}$ comes from. What is to be noticed is that for any $\lambda \in \mathbb{C}$, $\rho_{\lambda \psi}=\rho_{\psi}$. (You can see this for yourself by direct substitution).  That is, there is an uncountably infinite ambiguity when one says that states of a quantum system are the kets; and, they're not. The states of a quantum system are the special type of self adjoint operators as we defined above. In the literature, they are also called density matrices. Mixed states are represented by linear combinations of such density matrices with the appropriate coefficients.
