Introduction
Let us consider a Hamiltonian $\hat{H}$ for a certain system and suppose that I would like to know if it is possible to define it (i.e. its domain) in such a way that it results self-adjoint. In principle, computation of the deficiency indices and the use of the relative Von Neumann theorem should be enough to establish whether or not $\hat{H}$ is self-adjoint or if self-adjoint extensions exist. Nevertheless, in order to compute the indices, it is necessary to solve two differential equations.
Now, suppose my operator $\hat{H}$ - and so the adjoint $\hat{H}^{\dagger}$- is so complicated that it is impossible to find solutions of the above-mentioned differential equations.
In order to proceed in the task, a possibile way could consist in the study of the time evolution operator generated by $\hat{H}$ itself.
In particular it should be possible to write, at the first order in t, what follows, e.g. in position representation: \begin{equation} \langle{x}| \hat{U}_t |{\psi} \rangle=\int _{\mathbb{R}} dx' \langle{x}| \hat{U}_t | {x'}\rangle \langle{x'}| {\psi} \rangle \approx \langle{x}|{\psi}\rangle-\frac{\mathcal{i}}{\hbar} t \int _{\mathbb{R}} dx' \langle{x}| \hat{H} |{x'} \rangle \langle{x'}|{\psi}\rangle \end{equation}
Clearly this expression is useful only if the "matrix element" $\langle{x}| \hat{H} |{x'} \rangle$ is known.
Let us suppose that this is the case. Since the unitarity condition on $\hat{U}_t$ implies for the kernel of the integral the condition: \begin{equation} \biggl(\langle{x}| \hat{H} |{x'}\rangle \biggr)^{*}=\langle{x'}| \hat{H} |{x} \rangle, \end{equation} to test this property should be enough to establish if $\hat{H}$ is self-adjoint or not.
Question(s)
I pretty much believe that what I derived above is roughly correct. Nevertheless what puzzles me is that in the above procedure it is not clear to me where the domain in which I define $\hat{H}$ comes into play and, ultimately, beside the explicit form of the operator, is the domain that plays a crucial role with respect to the self-adjointness property.
So, in the end, I guess my questions are: what am I missing to complete/correct my demonstration in order to take into account the domain of $\hat{H}$? And how this domain is related to the domain of the relative operator $\hat{U}_t$? Is it possible the that kernel I am dealing with is not well-defined since it involves position "eigenstates"?