Is the operator $P=-i\hbar\frac{d}{dx}$ self-adjoint given the Hilbert space of the problem of particle in a box? The operator $P=-i\hbar\frac{d}{dx}$, is a symmetric operator in the domain $$D(P)=\left\{f(x) \big|f\in L_2[0, a], f(0)=f(a)=0\right\}$$ i.e. the domain is the subspace of square-integrable functions in the interval $0\leq x\leq a$ that vanish at the endpoints $x=0$ and $x=a$. The adjoint turns out to be $P^\dagger=i\hbar\frac{d}{dx}$.
The operator $P$ is symmetric but not self-adjoint because $D(P^\dagger)\neq D(P)$ where $$D(P^\dagger)=\left\{g(x) \big|g\in L_2[0,a]\right\}$$
If we want to apply this idea to the problem of a quantum particle in a box (infinite square well), then the wavefunctions representing all possible states of the system are both square-integrable and vanish at endpoints. So $-i\hbar\frac{d}{dx}$ is automatically self-adjoint. We don't need a self-adjoint extension. Am I right?
 A: You forgot to include the differentiability condition for the domain of $P$ and $P^\dagger$. Indeed, suppose our Hilbert space is $L^2(I)$ with $I:=[0,2\pi]$.$^\ddagger$ Define the domain of $P:=-i\frac{\mathrm d }{\mathrm dx}$ as
$$\mathcal D(P):=\{\psi\in C^1(I) |\psi(0)=0 = \psi(2\pi)\} \quad .$$ One can show that
$$\mathcal D(P^\dagger)= H^1(I) \quad , $$
where $H^1$ denotes the Sobolev space. It holds that $C^1(I)\subsetneq H^1(I)\subsetneq L^2(I)$. As such, we have $\mathcal D(P) \subsetneq \mathcal D(P^\dagger)$ and thus $P$ is not self-adjoint.

$^\ddagger$ It does not make sense to define the Hilbert space as something like $$\tilde L^2(I) := \{\psi\in L^2(I)|\psi(0)=\psi(2\pi) = 0\} \quad ,$$ because actually we are dealing here with equivalence classes of wave functions, which cannot see points of measure zero. Rather, the infinite potential at the boundaries of the box is taken into account by the domain of the Hamiltonian.
References and further reading:

*

*F.P. Schuller. Lectures on Quantum Theory. Lecture Notes, chapter 9. Here is a video of the relevant lecture: YouTube.


*The classic PSE thread: What's the deal with momentum in the infinite square well?
A: The correct answer to your question has already been posted, but I wanted to supplement that by noting a particular misconception:

If we want to apply this idea to the problem of a quantum particle in a box (infinite square well), then the wavefunctions representing all possible states of the system are both square-integrable and vanish at endpoints.

The Hilbert space for a particle in a box of length $a$ is $L^2\big([0,a]\big)$. There is no requirement that the wavefunctions vanish at the endpoints, and such a requirement in fact cannot even be imposed in the first place for two reasons.

*

*Despite the typical description, $L^2\big([0,a]\big)$ is not the space of square-integrable functions from $[0,a]\rightarrow \mathbb C$. Rather, it is the space of equivalence classes of such functions under the equivalence relation $\psi\sim \phi \iff \psi = \phi$ almost everywhere. Two functions which differ by their values at a single point represent the same element of $L^2\big([0,a]\big)$, and as a result we cannot impose any constraints which require wavefunctions to take a specific value at individual points, unless we also impose at least the requirement of continuity (or something stronger, like differentiability).


*Part of the definition of a Hilbert space is that it be topologically complete. This means that every Cauchy sequence $\{\psi_n\}$ must converge to a limit which is contained in the Hilbert space.  However, notice that
$$\psi_n(x) = \left[\frac{2x(a-x)}{a}\right]^{1/n}$$
is a Cauchy sequence, and that it converges (in the topology defined by the norm) to $\psi(x) = 1$:

