I know that the momentum operator $p$ defined on the Schwarz function $S(\mathbb{R})$ is essentially self-adjoint. However, what if I were to restrict $p$ to $C_c^\infty (0,1)\subseteq L^2(0,1)$. In this case, would $p$ still be essentially self-adjoint?
Without thinking too deeply, I think this is true since we could repeat the usual proof involve Fourier transforms, but I'm not completely sure, since I know that there isn't a self-adjoint extension on the half-line $\mathbb{R}_+$.
No, $\hat p$ is no longer essentially self-adjoint in this case. One problem is that essential self-adjointness is a property of symmetric/Hermitian operators, and the operator $\hat p_0: f \mapsto -if'$ with domain $C^\infty([0,1])$ is not Hermitian, which can be seen straightforwardly. The problem is that when we integrate by parts on a compact interval, we get boundary terms which don't generally vanish; in other words, the domain of $\hat p_0$ is too large.
In order to obtain a Hermitian operator, we need to restrict the domain by adding boundary conditions, e.g. the Dirichlet condition $f(0)=f(1)=0$. The operator $\hat p$ with domain $$\mathrm{dom}(\hat p):= \big\{f\in C^\infty([0,1]) \ : \ f(0)=f(1)=0\big\}$$ is Hermitian, and so we can now proceed to ask about essential self-adjointness. One way to proceed is by computing the so-called deficiency indices$^\ddagger$, defined as $$d_{\pm} = \mathrm{dim}\left(\mathrm{ker}\left(\hat p^\dagger \mp i\mathbb I\right)\right)$$ Once we have them, there are three possibilities:
The first thing to do is to compute $\hat p^\dagger$. For any $f\in \mathrm{dom}(\hat p^\dagger)$ and $g\in\mathrm{dom}(\hat p)$, there exists some $\psi_f\in L^2([0,1])$ such that $$\langle f, \hat p g\rangle = \langle \psi_f, g\rangle $$ $$\iff \int_0^1 \overline{f(x)}\big(-ig'(x)\big) \mathrm dx = \int_0^1 \overline{\psi_f(x)} g(x) \mathrm dx$$ It's not hard to see that this implies that $\psi_f$ is the (weak) derivative of $-if$. As a result, $p^\dagger: f \mapsto -if'$ and $\mathrm{dom}(\hat p^\dagger)=H^1([0,1])$, essentially the space of weakly differentiable functions whose derivatives are in $L^2([0,1])$ (see Sobolev spaces).
From here, computing the deficiency indices is easy. $$\mathrm{ker}(\hat p^\dagger\mp i\mathbb I):= \big\{f \in H^1([0,1]) \ : \ -if' \mp i f = 0 \iff f' \pm f = 0\big\}$$ $$\implies \mathrm{ker}(\hat p \mp i\mathbb I) = \big\{a e^{\mp x} \ : \ a\in \mathbb C\big\}$$ It's easy to see that both of these spaces are one-dimensional, so $d_+=d_-=1$. As a result, $\hat p$ is not essentially self-adjoint, but rather has an entire family of self-adjoint extensions $\hat p_\theta$. One can further show that their domains are restricted to include the quasiperiodic boundary conditions $f(1)=f(0)e^{i\theta}$ for $\theta\in[0,2\pi)$.
$^\ddagger$To see how these quantities arise from the Cayley transform, see e.g. here.
