Why do we represent states vectors with ket vectors? From what I currently understand given a general state vector $|\psi\rangle$ the wave function:
$$\psi(x) = \langle x|\psi\rangle$$
represent the vector $|\psi\rangle$ in the base of the eigenvalues of the position operator. Similarly the wave function
$$\psi(p)=\langle p|\psi\rangle$$
represent the same vector but in the base of momentum. In practice we can think of wave functions as column vectors with an infinite number of entries, one for every real number.
So when we write $|\psi\rangle$ do we mean to represent the abstract vector $|\psi\rangle$ without referring to a specific base? Why do we do this? In friendly 3D linear algebra we almost always think of vectors in the context of a specific representation of them in some base. Wouldn't be easier to always represent states vectors in some specific base, so as wave functions? I am saying this because using this double way of representing vectors sometimes tends to make things confusing; for example: in QM lectures happens often that a certain operator is described as acting on ket vectors:
$$A|\psi\rangle$$
and then after a bit the same operator, without any further explanation, is shown as acting on functions:
$$A\psi(x)$$
But there are some operations that make sense only if applied on functions and not on ket vectors. Why do we represent things in such a way? Why don't we only use wave function representation of vectors in some specific base?
 A: 
In QM lectures happens often that a certain operator is described as acting on ket vectors $A|\psi\rangle$ and then after a bit the same operator, without any further explanation, is shown as acting on functions $A\psi(x)$.

This is not correct.  You may have seen it somewhere, but the author was being sloppy or abusing notation.

Let $|\psi\rangle$ be an abstract ket vector. If we wish to represent it in the continuous position basis, we can insert the identity operator $\mathbb 1=\int|x\rangle\langle x| dx$ and obtain
$$|\psi\rangle = \int|x\rangle\langle x|\psi\rangle dx = \int |x\rangle  \psi(x) dx$$
Loosely speaking, $\psi(x)$ is the component of $|\psi\rangle$ along the basis vector $|x\rangle$.  If you want to think of something as an infinitely long column vector, then it should be $|\psi\rangle$, not $\psi(x)$ (which is just a complex number).
Similarly, if $A$ is an abstract operator, then we can let it act on abstract kets $|\psi\rangle$ as $A|\psi\rangle$.  Expanding $|\psi\rangle$ out in the position basis, we find
$$A|\psi \rangle = \int A|x\rangle \psi(x) dx$$
$A$ is still an abstract operator which acts on a ket (in this case, $|x\rangle$), not a function.  If we insert another identity operator $\int |y\rangle\langle y| dy$, we find
$$A|\psi\rangle = \iint |y\rangle\langle y|A|x\rangle \psi(x) dy \ dx$$
The object $\langle y|A|x\rangle \equiv A_{yx}$ is the $yx$ component of the abstract operator $A$.  This object is what acts on functions.  The result is that
$$A|\psi\rangle = \iint |y\rangle A_{yx} \psi(x) dy\ dx$$
For example, the position operator $Q$ has components $Q_{yx} \equiv \langle y|Q|x\rangle = \delta(y-x) \cdot x$ while the momentum operator has components $P_{yx} \equiv \langle y |P|x\rangle = -i\hbar \delta(y-x) \frac{d}{dx}$.  We would therefore have
$$Q|\psi\rangle = \int |x\rangle x\cdot \psi(x) dx$$
$$P|\psi\rangle = \int |x\rangle (-i\hbar)\psi'(x) dx$$

If we are being very strict, we would say that the position operator $Q$ eats a ket with position-space wavefunction $\psi(x)$ and spits out a ket with position-space wavefunction $x\psi(x)$.  However, we often relax a bit and say that $Q$ eats a wavefunction $\psi(x)$ and spits out $x\psi(x)$.
The reason we use kets in the first place is that it can be quite convenient to not restrict yourself to a particular basis. I find it very difficult to believe that you've never used the vector notation $\vec r$ as opposed to the index notation $r_i$, and this is precisely the same thing.  The only difference is that the index $i$ in $r_i$ runs over $\{1,2,3\}$, while the index $x$ in $\psi(x)$ runs over $\mathbb R$.
