Operator on infinite dimensional Hilbert space: domain and range Since a QM operator is a linear map, it is useful to think about them as functions.  An operator $\hat A$ on a finite $N$-dimensional Hilbert space $H_N$ is always such that
$$\hat A:H_N\to H_N.$$
The domain and range of $\hat A$ are both $H_N$.  $\hat A$ operates on states in Hilbert space and returns states in Hilbert space.  My question regards the range of an operator $\hat B$ on an infinite dimensional Hilbert space $H_\infty$.  If an operator always returns an eigenstate, we must write
$$\hat B:H_\infty\not\to H_\infty,$$
because the eigenstate of an operator with a continuous spectrum cannot live in Hilbert space.  The range cannot be the domain.  So, what is the function notation $\hat B:H_\infty\to X$ appropriate for operators with continuous spectra?
Regarding the physics, I want to know about the mechanism by which the position operator can operate on a state to kick it out of the Hilbert space by returning a Dirac $\delta$ position eigenstate.  I am aware that the rigged Hilbert space formalism offers a space for the position eigenstate to live in, but I am curious as to how this is described in the usual theory of linear operators on Hilbert space.
 A: The appropriate generalisation of eigenvalues of an operator $A$ on an infinite-dimensional Hilbert space $\mathcal{H}$ is the spectrum $\sigma(A)$ consisting of those $\lambda \in \mathbb{C}$ for which $\lambda - A$ does not admit a bounded inverse. The spectrum of a self-adjoint operator A with domain of self-adjointness $D(A)$ dense in $\mathcal{H}$ can be split into two parts: The discrete spectrum and the essential (or continuous) spectrum. The discrete spectrum consists of all isolated eigenvalues in $\sigma(A)$ with finite multiplicity, the essential spectrum of all other spectral values. If $\lambda$ lies in the spectrum, then, by Weyl's criterion, a sequence $(\psi_k)_{k\in\mathbb{N}}$ of normalised elements in $D(A)$ exists such that $\| A\psi_k - \lambda \psi_k\| \to 0$ as $k\to \infty$ (i.e. $A\psi_k$ approximates $\lambda \psi_k$). If $\lambda$ is an eigenvalue, simply take the constant sequence that consists of an eigenvector. If $\lambda$ lies in the essential spectrum, then $(\psi_k)_{k\in\mathbb{N}}$ can be chosen to have no convergent subsequence. Sometimes such a sequence is then called an approximate eigenvector.
Example: Let $X: D(X) \subset L^2(\mathbb{R}) \to L^2(\mathbb{R})$ be the position operator defined by $X\psi(x) = x\psi(x)$ for $\psi \in D(X)$, where $D(X)$ is the domain of $X$ that makes $X$ self-adjoint. It is known that $\sigma(X) = \mathbb{R}$. Let $x_0 \in \sigma(X)$. Naively, $\delta_{x_0}$ is an eigenfunction of $X$ with eigenvalue $x_0$ because
$$X\delta_{x_0}(x) = x \delta_{x_0}(x) = x_0 \delta_{x_0}(x).$$
However, $\delta_{x_0} \notin D(X)$ (and not even in $L^2(\mathbb{R})$). But $\delta_{x_0}$ can be approximated by normalised functions $f_\epsilon \in D(X)$, e.g.
$$|f_\epsilon(x)|^2 = \frac{1}{\sqrt{2\pi\epsilon}} \mathrm{e}^{-\frac{(x-x_0)^2}{2\epsilon}}.$$
Indeed, Weyl's criterion is satisfied because
$$\| Xf_\epsilon - x_0f_\epsilon\|^2_{L^2} = \frac{1}{\sqrt{2\pi\epsilon}} \int_{\mathbb{R}} (x-x_0)^2 \mathrm{e}^{-\frac{(x-x_0)^2}{2\epsilon}} \mathrm{d}x \stackrel{\epsilon \to 0}{\longrightarrow} 0. $$
