How should Dirac notation be understood? If vectors $|\vec{r}⟩$ and $|\vec{p}⟩$ are defined as 
$$
\hat{\vec{r}} |\vec{r}⟩ = \vec{r} |\vec{r}⟩ \\
\hat{\vec{p}} |\vec{p}⟩ = \vec{p} |\vec{p}⟩
$$
then one can see that products like
$$
⟨\vec{r}|\vec{r}⟩ \\ ⟨\vec{p}|\vec{p}⟩
$$
do not converge, which means that $|\vec{r}⟩$ and $|\vec{p}⟩$ do not belong to a Hilbert space as it is necessary that $\int |\psi|^2 < \infty$. So how should one understand these objects and what space do they belong to?
 A: Physicists usually generously relax the condition that the norm should be finite and they sometimes say that $|\vec r\rangle,|\vec p\rangle$ belong to the "Hilbert space". It's exactly the same "generous" language that allows physicists to say that $\delta(x)$ is a "function", the delta-function, even though its values around $x=0$ are infinite or "ill-defined", so it's not really a function.
Rigorously mathematically, those objects don't belong to the Hilbert space (because their norm isn't finite) but (in analogy with the concept of "distributions" that include "functions" like $\delta(x)$) there exists a mathematical concept that includes such non-normalizable vectors, the rigged Hilbert space.

https://en.wikipedia.org/wiki/Rigged_Hilbert_space

The idea that is that one may define a subspace $\Phi$ of the Hilbert space that contains smooth enough functions. The dual space to the Hilbert space $H$ is $H$ itself. But the dual space to the subspace $\Phi$ is $\Phi^*$ that is, on the contrary, larger than $H$, and it's this $\Phi^*$ that is called the rigged Hilbert space and that contains objects such as $|\vec r\rangle$. Formally mathematically, the whole triplet $(\Phi^*,H,\Phi)$ is usually referred to as "the rigged Hilbert space".
A: Just to add to what's been said and to address the comment by @LightnessRacesinOrbit, you can indeed have perfectly good functions that are only defined on some subset of the real numbers.  The reason the Dirac delta is not a function is not that it is only well-defined for some real numbers and not others; in fact, it is not defined as a function anywhere!  It is instead defined as a sort of "hypothetical" function, call it $\delta(x)$, that, if it existed, would satisfy the property that when you integrated it against any other (sufficiently nice) function $f(x)$, you would get
$\int_{\mathbb{R}} \delta (x) f(x) \, dx = f(0) $.
There is no well-defined function that actually does this, but there are well-defined functions that come as close as you want:  for example, functions we can call $\varphi_\varepsilon$ that are equal to zero everywhere except in a small interval  $(-\varepsilon, \varepsilon)$ around zero, and are equal to $\frac{1}{2\varepsilon}$ for $x \in (-\varepsilon, \varepsilon)$.  Notice that if $f(x)$ is smooth, then 
$\int_{\mathbb{R}} \varphi_\varepsilon (x) f(x) \, dx \approx f(0) $.
So we can get close to having a function that behaves the way we want a Dirac delta to, but not quite (that equality is only approximate, because $f(x)$ may vary over the interval $(-\epsilon,\epsilon)$).  Rather than a function, the Dirac delta is really a linear operator that takes in a smooth function and gives you back the value of that function at $x=0$.  Notationally we still pretend that there is a "function" $\delta(x)$ that embodies this operation when you integrate other functions against it, but more properly it is an element of the so-called "dual space" to the space of smooth functions that are zero outside a bounded interval - that is, it is an element of the space of linear operators on such functions (as a technicality:  note that it is a continuous linear operator with respect to the uniform norm on smooth functions of compact support).
