What subset of the Hilbert space is a physical state? Not all Hilbert space vectors are generally considered physical, due to various reasons. A particular example (as found in Hall's "Quantum mechanics for mathematicians") is the classic particle in a box example : only states obeying the boundary conditions $\psi(0) = \psi(L) = 0$ are considered.
He later mentions that states not obeying this condition are not part of $\text{Dom}(\hat H)$, which I'm guessing might be the theoretical ground for dismissing them.
Is there a generic criterion for qualifying which states are physical? To which operators should they be part of the domain to qualify as such?
 A: Consider the free particle on the real line with standard Hamiltonian $$H = \frac{1}{2m}P^2\:.$$
Next consider a state  $\psi=\psi(x)$ consisting of a compactly-supported smooth function attaining constantly the value  $c\neq 0$ in the interval $[a,b]$ and smoothly  vanishing outside that interval. 
$\psi$ seems a good candidate for a physically meaningful wavefunction. I stress that it simultaneously belongs to the domains of $H$, $X$, and $P$.
(The Fourier transform $\hat{\psi}$ of $\psi$ belong in the Schwartz space so that $\psi$ is an element of $D(P^k)$ for every $k=1,2,\ldots$)
Now perform a measurement of position and suppose that the particle is found in $[a',b'] \subset [a,b]$. 
In view of that standard projection postulate the wavefunction immediately after the measurement is 
$$\psi'(x) = Nc \quad\mbox {if $x\in [a',b']$, $\quad \psi(x) =0\quad$ if $x\not\in [a',b']$}$$
where $N$ is the normalization constant.
This new function does not belong  to the domain of $H$. Indeed,
$$D(H) = \left\{\psi \in L^2(\mathbb R)\:\left|\: \int_{\mathbb R} p^4|\hat{\psi}(p)|^2 dp <+\infty\right.\right\}$$
In the considered case, for some positive constant $C$,
$$p^4|\hat{\psi'}(p)|^2  = C p^2 (1-\cos(p(b'-a'))\:.$$
The integral of this function diverges (I used an online integrator to be sure completely). Thus $\psi' \not \in D(H)$.
This elementary and  very idealized example shows how the existence of incompatible observables ($X$ and $H$ here) and the collapse postulate make very difficult to define a class of physically sensible states as belonging to a suitable class of domains of physically meaningful observables.
There are actually several ways out from my no-go example. One may argue that the collapse postulate is oversimplified and physical measurement procedures of continuous spectrum observables admit a more accurate description in terms of POVM and quantum operations. This is true, my intention was just to show how the problem is subtle.  
A: A physical (pure) state is the full unit ray of any solution to the Schroedinger equation:
$$ \frac{d\Psi (t)}{dt} = \frac{1}{i\hbar} H(t) \Psi (t) $$
subject to an initial condition $\Psi (t=0) = \psi_0 $, where all psi and Psi are unit norm vectors in a complex separable Hilbert space. This equation is readily solved for a time-independent Hamiltonian and the particular set of states called https://en.wikipedia.org/wiki/Stationary_state. Schroedinger's equation is reduced to the spectral equation for the Hamiltonian operator which can very well have no solution in the Hilbert space, for example for a free Galilean particle of spin=0 in $\mathbb{R}^n$. By definition, the set of eigenstates of the time-independent Hamiltonian is a subset of the domain of self-adjointness of H. 
