Since a QM operator is a linear map, it is useful to think about them as functions. An operator $\hat A$ on a finite $N$-dimensional Hilbert space $H_N$ is always such that $$\hat A:H_N\to H_N.$$
The domain and range of $\hat A$ are both $H_N$. $\hat A$ operates on states in Hilbert space and returns states in Hilbert space. My question regards the range of an operator $\hat B$ on an infinite dimensional Hilbert space $H_\infty$. If an operator always returns an eigenstate, we must write $$\hat B:H_\infty\not\to H_\infty,$$
because the eigenstate of an operator with a continuous spectrum cannot live in Hilbert space. The range cannot be the domain. So, what is the function notation $\hat B:H_\infty\to X$ appropriate for operators with continuous spectra?
Regarding the physics, I want to know about the mechanism by which the position operator can operate on a state to kick it out of the Hilbert space by returning a Dirac $\delta$ position eigenstate. I am aware that the rigged Hilbert space formalism offers a space for the position eigenstate to live in, but I am curious as to how this is described in the usual theory of linear operators on Hilbert space. If you have a link, I'd like to read about this generally more so than I want any one thing clarified. Thanks.
