"Dot product" of kets from different spaces? In the class, my lecturer claimed that $|p\rangle$ and $|x\rangle$ don't live in the usual Hilbert space like the state vector $|\psi\rangle$, but in their own spaces called "position space" and "momentum space", so the operation
$$
\begin{equation}
\langle{x|\psi}\rangle = \psi(x)
\end{equation}
$$
and
$$
\begin{equation}
\langle{p|\psi}\rangle = \psi(p)
\end{equation}
$$
are not inner product because one cannot perform such operation for vectors in different space. Hence we have to define them as $\psi(x)$ and $\psi(p)$, instead of a scalar quantity.
It seems like there's an operation that maps a vector from position/momentum space with a vector in Hilbert space back to the Hilbert space. I am sceptical about this statement. Can anyone verify does such an operation exists? If so, what is it called? And if not, what goes wrong in the statement?
 A: Both, position or momentum, can form a basis for the space of $|\psi\rangle$.
The object that can be converted between "spaces" is not the state but the representation of the state. I.e. you have the representation with respect to the position basis which is $\psi(x)$ and then you have a representation with respect to the momentum basis, which is $\tilde \psi(p)$. Both representations belong to the same state $|\psi\rangle$,
$$
|\psi\rangle = \int^\infty _{-\infty}dx \psi(x)|x\rangle =  \int^\infty _{-\infty}dp \tilde \psi(p)|p\rangle 
$$
The representation $\psi(x) $ and $\tilde \psi(p)$ in 1 dimension are connected via
$$
\psi(x) = \frac{1}{\sqrt{2\pi\hbar}}\int^\infty _{-\infty}dp \ \tilde \psi(p)\exp(ipx/\hbar ) \\
\tilde \psi(p) = \frac{1}{\sqrt{2\pi\hbar}}\int^\infty _{-\infty}dx \  \psi(x)\exp(-ipx/\hbar ) \\
$$
This transformation pair can be defined differently up to constants and the sign in the exponential, so make sure to always state the transformation/convention that you are using. $\psi(x)$ and $\tilde\psi(p)$ have different units and it would not make sense to add them. These objects are said to live in position and momentum space since they are space amplitudes and momentum amplitudes. This transformation pair is an example of a Fourier transform. But let me repeat this, the transformation definition contains conventions which should always be stated clearly !
The tricky thing about continuous bases $|x\rangle, |p\rangle$ is that a single basis vector is not a valid physical state, since it is not normalizable, only sums of these states that are weighted such that their magnitude square integral converges, are in the set of physically admitted states. So don't try to overinterpret a single $|x\rangle$ or a single $|p\rangle$ state, only states of the form $\psi(x)|x\rangle$ or $\tilde \psi(p)|p\rangle$ such that the integral $\int |\psi(x)|^2dx$ or $\int |\tilde \psi(p)|^2dp$ converges, are physically admissible.
Answer to comment:
The inner products $\langle  x|p\rangle, \langle  p|x\rangle$ allows you to transforms expression from one representation to the other. If you use the resolution of identity
$$1 = \int^\infty_{-\infty}dx \ |x\rangle \langle x| = \int^\infty_{-\infty}dp \ |p\rangle \langle p|$$
you typically end up with inner products which need to be evaluated. For example $$|\psi\rangle=\int^{\infty}_{-\infty}dx \ \psi(x)|x\rangle= \int^{\infty}_{-\infty}dx \  \psi(x)1|x\rangle= \int^{\infty}_{-\infty} dx \ \psi(x)\int^\infty_{-\infty}dp \ |p\rangle \langle p|x\rangle=\int^\infty_{-\infty}dp \left( \int^\infty_{-\infty}dx \langle p|x\rangle \psi(x) \right) |p\rangle = \int^\infty_{-\infty}dp \tilde \psi(p) |p\rangle $$
The integral
$$
\left( \int^\infty_{-\infty}dx \langle p|x\rangle \psi(x) \right) = \tilde \psi(p)
$$
allows you to calculate the momemtum wavefunction starting from the position wavefunction and if we compare it with my former definition, we see that ,
$$
\langle p|x\rangle = \frac{1}{\sqrt{2\pi\hbar }}\exp(-ipx/\hbar)
$$
The question where $|x\rangle, |p\rangle$ exactly live is beyond me and involves more advanced math than i am familiar with. All that i know for certain is that the objects $\psi(x)|x\rangle, \ \tilde \psi(p)|p\rangle$ are elements of the physically relevant Hilbert space, which is not true for a single $|x\rangle, |p\rangle$.
Someone who is more familar with Gelfand Triples and the likes might be able to give a better answer to what exactly these objects by themselves are.
