Furthermore, neither of these functions has eigenvectors in $\mathcal{H}$: if $X\,f(x) = \lambda f(x) = x f(x)\,\forall x\in\mathbb{R}$ then $f(x) = 0$ for $x\neq\lambda$ and the eigenfunction $e^{-i\,k\,x}$$e^{i\,k\,x}$ of $P$ is not normalisable.
For QM we take the dense subset $S$ to be the "smooth" functions that still belong to $\mathcal{H}$ when mapped by any member of the algebra of operators generated by $X$ and $P$. That is, $S$ is invariant under this algebra and comprises precisely the Schwartz space of functions than can be multiplied by any polynomial and differentiated any number of times and still belong to $\mathcal{H}$. Any function in $\mathcal{H}$ can be arbitrarily well approximated (with respect to the Hilbert space norm) by some function in $S$.
At the same time, we kit the dense subset $S$ out with a stronger topology than the original Hilbert space one. Why do we do this? One of the basic problems with $\mathcal{H}$ is that the Dirac delta $\delta:\mathbf{L}^2(\mathbb{R})\to \mathbb{C};\;\delta\;f(x) = f(0)$, which can be construed as an eigenvector of $X$, is not a continuous linear functional on $\mathcal{H}$ even though of course it is a linear functional. To see this, consider the image of $f(x) + \exp(-x^2/(2 \sigma^2)$ under the delta funcional: we can choose a $\sigma$ to make this function arbitrarily near to $f(x)$ as measured by the $\mathbf{L}^2$ norm, but with images $f(0)$ and $f(0)+1$, respectively, under the Dirac $delta$$\delta$. So we kit the dense subset $S$ out a topology that is strong enough to "ferret out" all useful linear functionals and make them continuous. We now have a topological dual (space of all linear functionals continuous with respect to the stronger topology) $S^*$ of $S$ such that $S\subset\mathcal{H} = \mathcal{H}^*\subset S^*$.