It usually just means that the expectation of the energy of the state is infinity (and so could never arise from a state with finite average energy by time evolution or measurements), so that the physical state space does not need these states, except as limits.
An example for P is the state
$$\psi(p) = {1\over\sqrt{p^2+1}} $$
This is normalizable (i.e., square integrable), but multiplying by $p$ takes you out of the normalizable states---so it takes you outside the Hilbert space---so $\psi$ is not, strictly speaking, in the domain of P. The Hamiltonian has a P$^2$ term, so that the expectation of the energy of this state is infinite, since
$$ \langle \psi |P P|\psi\rangle =\infty$$
means that $\langle P^2\rangle=\infty$, so that $\langle H\rangle=\infty$.
An analogous $\psi$ for x would be defined by replacing $p$ above by $x$: $$\phi(x) = {1\over\sqrt{x^2+1}} $$ Such a state is not as localized, even though zero is its average position. But it has an infinite variance in position--- if you measure the position again and again, you will not have an average deviation. This wavefunction is infinitely spread out, so it is not really useful for describing a particle which is somewhere. It is an idealization, like a plane-wave state.
These mathematical subtleties are never all that interesting outside of pure mathematics--- they are eliminated completely by working in a finite lattice of a bounded size, and this procedure must preserve all physical behavior for a large enough lattice with small enough spacing. The regulator cannot wreck the physics.
See also here: Regularisation of infinite-dimensional determinantsRegularisation of infinite-dimensional determinants