Are states of a quantum system represented by vectors in a Hilbert space?

Is it true that the states of a quantum system are represented by vectors in a Hilbert space?

When does the term "ray" come into play in this context?

That's a rather delicate topic. I suggest you to begin with Section 2.1 ''Quantum Mechanics'' of Weinberg's ''The Quantum Theory of Fields'', Volume I.

All (normalized) wave vectors in the Hilbert space which only differ by phase represent the same physical state of the system. All such wave vectors (corresponding to the same state) can be united into a ray in order to have a one-to-one correspondence between rays and pure states of the system. (There also exist mixed states of the quantum system - those which have to be discussed in terms of a more general formalism of the density matrix...)

So, here's the plan:

1. You start with some vector space which you call the Hilbert space (with a positively defined scalar product).
2. You limit yourself with the subset of normalized wave vectors (you only use those when describing the physical states).
3. It turns out that those of them which only differ by phase represent the same state of the system. \begin{alignedat}{4} &|\psi\rangle \quad&&\to\quad &&\text{some state of the system}\\ \operatorname{e}^{i\,\alpha}&|\psi\rangle \quad&&\to\quad&&\text{same state of the system} \end{alignedat} where $\alpha$ is a real number.

Be mindful that even though the wave vectors representing the states should be normalized, during the calculations we often deal with the non-normalized ones. So, typically it's like that: do some calculations, normalize the result at the very end. However, one should remember that the operations of normalization and, say, addition do not commute.

Quantum Mechanical operations are all done in Hilbert Space. A Hilbert space is an infinite dimensional complex vector space. There are many reasons why we do so. For example, if you try to find the mechanics of body by using the Hamiltonian formulation in Classical Mechanics, you will study the object's flow in phase space. If you use use Lagrangian formulation the body's dynamics is seen in configuration space. But you should see that the system is unaffected by the formulation. The formulations differ only in their mathematical framework. Like wise in Quantum mechanics, you should realize that no observations are merely possible. I will explain why.
For a macroscopic body, you could observe or evaluate the body's properties like energy, momentum, position etc. without affecting the body's dynamics. You could make your measurement and tell the answer to any degree of accuracy you need. But when it comes to the microscopic particles, the observations or any try of making a measurement will change the system's properties. It is very fundamental property of quantum mechanical systems. Experiment like sequential Stern-Gerlach experiment tells us that. (You may also refer to the Quantum Double Slit Experiment, which is essentially a thought experiment.)
So this means you cannot measure any properties of a system at the microscopic level. Suppose you need to measure a particle's energy at this instant. If you make your observation, or experiment , that observation disturbs, more precisely alters the state of the system. So what you get is the result of your measurement, not the information that you wanted about the system , which is what we don't want.
Due to this, the story of quantum mechanics is formulated in pure theoretical framework with precise postulates. Any classical properties correspond to an observable in Quantum mechanics. An observable is an operator that corresponds to a physical quantity, such as energy, spin, or position, that can be measured; think of a measuring device with a pointer from which you can read off a real number which is the outcome of the measurement. So instead of taking direct measurements, we use operators in quantum mechanics. Also the operators should act on something. The energy operator, for example should give the energy of the system. So the function on which our operator need to be operated should be such that it contains information about what we seek. So we define some possible states corresponding to a system which it could occupy, by the assumption that the system can occupy any one state at a single time. Our mathematical function can now be expressed as a linear combination of these states. That's how we define a state ket (a mathematical abstract for what we call the wave function). Now as in the case of Lagrangian and Hamiltonian mechanics, we define a space made of axes with each dimension corresponding to these individual states. It's like representing a vector in Cartesian co-ordinates by three base vectors. You could write a vector as a linear combination of these base vectors (x^, y^, z^).
But there is an essential condition that to be met inorder to write a state ket like this. The base kets you select should be eigen kets of the observable you are choosing. The dimensionality depends on the system and upon what state you are corresponding to. For example, in the Stern-Gerlach experiment, the experiment measures the spin of the electron in the incoming beam of atoms. There by the system (the atoms) have two degrees of freedom- one corresponding to spin up and the other spin down. So the system in terms of spin can be described by two base kets (we get a two dimensional vector space).
However there are situations in QM where there is a need for infinite dimensions. So, say the QM lives in an infinite dimensional vector space called Hilbert Space.
Now let's deal with your question. You should be confused by the statement "we are dealing with rays in QM rather than vectors". It means a state ket |x> and a constant multiplied by the same, say, a|x> where a is a complex constant both corresponds to the same state. For example, if you think about force and acceleration, the vector force is nothing but a scalar (mass) times a vector (acceleration). A scalar times a vector gives you a new vector with magnitude different, but both vectors sharing a common direction. So in terms of directions, acceleration corresponds to force. It is true. Whenever we say that a body is accelerating we know there is a corresponding force acting on the system. Otherwise, the force and acceleration correspond to the same state of the system.
That's why we say we are concerned with rays rather than vectors.
This has something to do with base kets while defining the state vector. I have mentioned that the state ket can be expanded as a linear combination of base kets which are eigen kets of an observable. While operating a state ket, say |x>, by an operator A, in general the new ket may not be proportional to the initial ket |x>.
But for an operator there are some kets say, |a'>, |a''>, |a'''. etc. so that the operator acing on the ket reproduces the same ket by a multiplicative constant. In such a case the kets |a'>, |a''>, |a'''. etc. are called eigen kets of the observable A. i.e., A|a'>=a'|a'>; A|a''>=a''|a''>; etc.
where a', a'' are the corresponding eigen values of the eigen kets.
In QM we are concerned only about such kets as they reproduce the initial ket. Our theoretical framework was such that the state ket should be unaltered by our observation or measurement of a property. While dealing with eigen kets of an operator, we could find the corresponding value of the observable without changing the state of the ket. This is the same as what we have said "in Qm we ware concerned only about rays rather than vectors".
Due to this reason, we will expand our state ket in terms of the eigen kets (the base kets of Hilbert space where we define our state ket).
See for reference: Modern Quantum Mechanics by J.J.Sakurai

• A Hilbert space is not infinite dimensional, a Hilbert space has an additional property that always automatically holds for finite dimensional inner product spaces. So it's only the infinite dimensional inner product spaces that you have to worry about whether they are Hilbert or not. – Timaeus Apr 13 '16 at 2:12
• I have no quarrel with that. If there is something that you find misleading in my answer. If so please tell me. I will edit it if necessary – UKH Apr 13 '16 at 4:17
• Can you use Latex to format your formulas and use punctuation and blank lines to separate different parts of the answer, in such a way to improve readability and clarity of the same? – nbro Mar 11 '18 at 19:43
• At the time I posted this answer, I didn't knew latex. But I learned. See my latest answers if you are interested. – UKH Mar 14 '18 at 12:30