Why do we use vectors in quantum mechanics? I've been trying to make my understanding of quantum mechanics more mathematically rigorous, but I'm struggling a bit with the lack of intuition behind the fact that we represent quantum states with vectors. In the first chapter of Principles of Quantum Mechanics, R. Shankar gives readers an overview of the math behind  QM, and in the section about ket notation, he explicitly says that with time, a student will learn to drop the inclination to associate magnitude and direction with every vector. But in math (at least as far as I've studied: high school math with a little bit of linear algebra and geometric algebra), the definition and usage of vectors largely revolves around the fact that they have magnitude and direction. So  if we don't associate these two key qualities with vectors in quantum mechanics, why do we use the terminology?
I tried to answer that to a certain extent: a lot of the operations we use are similar: inner products, scalar products, and so on. But we don't even use the conventional vector notation (for example dot products for scalar products are written as $\langle\phi|\psi\rangle$). Similarly, we use the mathematical properties of eigenfunctions for stationary states, but the intuition of changing the modulus but not the direction of the vector doesn't seem obvious to me in the context.
Is there a deeper reason why we use vector terminology? Perhaps certain historical things which evolved from matrix mechanics? (I haven't rigorously studied matrix mechanics)
 A: The terminology "vector" behind a quantum state is justified due to the fact that quantum states are elements of a Hilbert space $\mathcal{H}$ (which is a vector space).
The inner product $\langle\psi|\phi\rangle$ is then the usual vector product in the following sense. Suppose that $|\phi\rangle\in\mathcal{H}$ and $\langle\psi|\in\mathcal{H}^*$, where $\mathcal{H}^*$ is the vector space dual to $\mathcal{H}$. 
$\mathcal{H}^*$ consists of all the linear functions $\langle\psi|:\mathcal{H}\to \mathbb{C}$, with the property that: 
$$\langle e^i|e_j\rangle=\delta^i_j$$
Here $\{e_j\}$ is a chosen basis for $\mathcal{H}$ and $\{e^i\}$ is a chosen basis for $\mathcal{H}^*$. These basis elements are the sets of eigenvalues of any chosen Hermitian operator.
A: 
But in math (at least as far as I've studied: high school math with a little bit of linear algebra and geometric algebra), the definition and usage of vectors largely revolves around the fact that they have magnitude and direction.

This is probably not obvious to you, but the key words in this sentence are "little bit" as regards linear algebra.
In full-grown mathematics, and particularly as regards linear algebra, vectors are not "things that have magnitude and direction". Instead, those concepts take a seat at the back of the bus, and we rephrase that concept as:

vectors are objects that satisfy the vector-space axioms.

This includes things like arrows-with-a-magnitude-and-a-direction in two or three dimensions, but - as it turns out - pretty much everything useful that you can say about arrows-with-a-magnitude-and-a-direction follows directly from the vector-space axioms (possibly augmented with the notion of an (abstract) inner product). And, because the way to make mathematics truly thrive is to make things as general as possible without sacrificing the results, the way we develop the mathematics for vectors is to work directly for vector spaces (i.e. any objects that satisfy the axioms), so that our results will be useful for arrows-with-a-magnitude-and-a-direction but also for a broad swathe of other objects.
What sort of other objects, you ask? Well, as a small selection:


*

*Arrows-with-a-magnitude-and-a-direction but in more than three dimensions, i.e. the space $\mathbb R^n$ of $n$-tuplets of real numbers. Which, if you really think about it, cannot really be assigned a "direction" in any truly understandable geometric terms.

*The same but with complex numbers: $\mathbb C^n$ works algebraically much the same as $\mathbb R^n$, so the same results should apply, but again it isn't really interpretable as an "arrow" with a "direction".

*Matrices, i.e. $\mathbb R^{m\times n}$, which again follow the same algebraic rules, with the same axioms and therefore the same consequences.

*Infinite sequences $\mathbb R^\infty = \{(x_1,x_2,\ldots) | x_j \in \mathbb R\}$.

*Function spaces, which again obey the same axioms, so they are also subject to the consequences of those axioms.


As regards quantum mechanics, very often we work in finite-dimensional spaces like $\mathbb C^n$, in which case the 'vector' language is maybe easier to digest, but the language that's bothering you is the use of the term 'vector' for something that lives in a function space like, say,
$$
L_2(\mathbb R) = \left\{\psi:\mathbb R\to \mathbb C | \int|\psi(x)|^2\mathrm dx < \infty \right\},
$$
where the use of the term 'vector' is simply because $L_2(\mathbb R)$ is a vector space as regards the vector-space axioms, which are simply the most useful way to characterize the behaviour of arrows-with-a-magnitude-and-a-direction.
A: This puzzled me as well when I started learning about QM.
The key insight is, indeed, that you have to stop thinking about vectors as objects with a direction and a magnitude. Well, quantum states do have a magnitude (and I guess you could associate a 'direction' with them), but it is not always useful to think of them in the same way as you would think of vectors as arrows on a piece of paper.
In the abstract definition preferred by mathematicians, a vector space is defined as a space of objects which (1) can be added together and (2) can be multiplied by scalars. 
Roughly speaking, these are the only two requirements. In mathematics, the notion of a 'vector' is not at all limited to the everyday intuition of arrows pointing some way, and includes all objects obeying the above requirements. You can make up many vector spaces which have nothing to do with the garden-variety vectors familiar from high school; many of these are useful in physics. The reason to think of all these as vector spaces is that mathematicians like to write down general theorems which hold for all vector spaces in general, or for certain large subclasses of vector spaces. This results in a multitude of tools available to everyone doing quantum mechanics. 
So the reason that we use vector spaces to describe quantum states is that quantum states also constitute a vector space in this sense: one can add them together and multiply them by scalars, as I think you noticed. 
An easy way to see that quantum states obey the requirements for a vector space is to toy around with wave-functions, which are quantum states written out in the position basis: it is obvious that these can be added together and multiplied by a scalar, although you do often have to worry about the normalization.
Another very useful idea from the theory of vector spaces is the use of a basis, which is a set of vectors, allowing every state to be expressed as a linear combination of basis vectors. One can decompose a quantum state into position eigenstates, energy eigenstates, momentum eigenstates or whatever one likes, using mathematical tools that are analogous to changing basis vectors in 3D. The idea of applying matrices to vectors is analogous to applying operators to quantum states, and turns out to involve similar mathematics as well.
The notion of an inner product is actually an extra structure which is not automatically included in every vector space. In QM, this extra structure of inner product also turns out to be very useful, because it allows one to take the norm of states and to calculate expectation values of operators, but keep in mind that not every vector space comes with an inner product.
A: You say that "we don't associate these two key qualities [magnitude and direction] with vectors in quantum mechanics". It is correct that the magnitude of the vector does not have a meaning, but the direction is very important. It is not a "direction" in the every-day sense of the word, since we are not talking about a 3D real vector space. But it is normal in math to use $n$-dimensional vector spaces, also complex vector spaces, and still call the objects "vectors".
Put differently, the best way to describe a quantum state is as a direction in an abstract vector space. Directions are commonly represented as normalized vectors in that space, that is what we do in quantum mechanics.
A: You should look into more linear algebra. In linear algebra, the notion of a vector becomes considerably more abstract: in particular, vectors are simply any objects for which we have a notion of adding two of them together as well as multiplying by a number (a scalar), that of course also satisfy certain familiar basic principles of arithmetic like commutativity and associativity that you should already know from your Algebra background. This of course derives ultimately from the "Euclidean vector" context, i.e. "arrows from the origin" that you can make longer and shorter and add together by placing their tail to their tip and forming the parallelogram, but is considerably more general because there are many other things which also work this way such as functions and even matrices themselves.
The reason for this in Quantum Mechanics is that you want to be able to form superpositions - the whole "Schrodinger's cat" business, which is absolutely fundamental and thoroughgoing at the quantum scale.
It would perhaps be best to walk through a sort of mathematical construction of a simple vector space of the type used in quantum mechanics to get a hang of what is going on.
We start of course with an empty set, $S := \emptyset$. Now, we insert into this set two elements which we will denote $|\mbox{live cat}\rangle$ and $|\mbox{dead cat}\rangle$. These two things are "primitive" objects - you shouldn't try to think of them as something with a "value" you can "evaluate". This is a common math failing; we are so used to, say, taking the decimal expansion of a real number and associating in our minds that this is the "truth" "behind" a symbol like $\pi$, for example, when actually it in reality has no more or less "truth" than something else that is equivalent, like $4\left(1 - \frac{1}{3} + \frac{1}{5} - \frac{1}{7} \cdots\right)$, i.e. Madhava's series - and if anything due to its simple pattern the latter is a far more preferable representation than the decimal! Rather they are just two objects, if you like the "meaning" behind them is what is on the label. What you have to do here is forget the tendency to "calculate" things.
But of course, we need more. What happens when the box is closed? We need something like $|\mbox{live cat}\rangle + |\mbox{dead cat}\rangle$. You know what that is. So we add that to $S$ as well.
However, we can have other combinations possible as well. For one, we mentioned the need to rescale vectors, so for $|\mbox{live cat}\rangle$ we should also add all scalings of it to the set: $\alpha |\mbox{live cat}\rangle$ for some complex number $\alpha$. These don't do anything physically, but they become important because they allow us to weight the combinations in a superposition. We can do the same for $|\mbox{dead cat}\rangle$ as well. If you go through all of these and consider various sums, e.g. $|\mbox{live cat}\rangle + 2|\mbox{dead cat}\rangle + (i|\mbox{dead cat}\rangle - [35 - \tau i]\mbox{live cat}\rangle)$, etc. and use the laws of arithmetic like associativity and combining like terms to simplify them (because you must be able to for this to make sense as an abstract vector space), you quickly see that every element should be, most generally, something like
$$\alpha|\mbox{live cat}\rangle + \beta|\mbox{dead cat}\rangle$$
and we can consider all those together, put into the set $S$, as defining the vector space for this system, which contains all possible ways that Schrodinger's cat can be superposed including both the classical live/dead states and all the "peculiar" states, and we can superpose any of them in any way we like. If we did not have this ability, we could not make sense of this system, or any other quantum system, for that matter.
