- Is the momentum operator $P=-i\hbar \frac{\mathrm d}{\mathrm dx}$ symmetric when restricted to the compact interval of the well? Are there any subtleties in its definition, via its domain or similar, that are not present for the real-line version?
(I henceforth assume the Hilbert space is $L^2([0,1],dx)$.)
It depends on the precise definition of the domain of $P$. A natural choice is $$D(P) = \{\psi \in C^2([0,1])\:|\: \psi(0)=\psi(1)=0\}\:.\tag{1}$$
With this definition $P$ is symmetric: (a) the domain is dense in $L^2([0,1], dx)$ and (b) the operator is Hermitian $$\langle P\psi| \phi \rangle = \langle \psi| P\phi \rangle\quad \mbox{for $\psi, \phi \in D(P)\:.$}\tag{2}$$
You can consider different definitions of the domain more or less equivalent. The point is that the self-adjoint extension are related with the closure of $P$ and not to $P$ itself, and you may have several possibilities to get the same closure starting from different domains. The situation is similar to what happens on the real line. There $P$ can be defined as a differential operator on $C_0^\infty(\mathbb R)$ or $\cal S(\mathbb R)$ (Schwartz' space), or $C^1_0(\mathbb R)$ and also interpreting the derivative in weak sense. In all cases the closure of $P$ is the same.
- Is the momentum operator $P=-i\hbar \frac{\mathrm d}{\mathrm dx}$ self-adjoint in these conditions? If not, why not, and what are the consequences in terms of the things we normally care about when doing one-dimensional QM?
It is not self-adjoint with the said choice of its domain (or with every trivial modification of that domain). The consequence is that it does not admit a spectral decomposition as it stands and therefore it is not an observable since there is no PVM associated with it.
Defining $P= -i \hbar \frac{\mathrm d}{\mathrm dx}$ on the real line with one of the domain said above, the same problem arises.
The general fact is that differential operators are never self-adjoint because the adjoint of a differential operator is not a differential operator since it cannot distinguish between smooth and non-smooth functions, because elements of $L^2$ are functions up to zero-measure sets. At most a symmetric differential operator can be essentially self-adjoint, i.e., it admits a unique self-adjoint extension (which coincides to the closure of the initial operator). This unique self-adjoint operator is the true observable of the theory.
Yes. The canonical way is checking whether defect indices of $P$ with domain (1) are equal and they are. But the shortest way consists of invoking a theorem by von Neumann:
If a (densely defined) symmetric operator commutes with an antilinear operator $C$ defined on the whole Hilbert space and such that $CC=I$, then the operator admits self-adjoint extensions.
In this case $(C\psi)(x):= \overline{\psi(1-x)}$ satisfies the hypothesis.
If it does, is that extension unique?
NO it is not, the operator is not essentially self-adjoint.
If the extension is not unique, what are the different possible choices, and what are their differences? Do those differences carry physical meaning / associations / consequences? And what is a self-adjoint extension anyways, and where can I read up about them?
There is a class of self-adjoint extensions parametrized by elements $\chi$ of $U(1)$. These extensions are defined on the corresponding extension of the domain
$$D_\chi(P) := \{\psi \in L^2([0,1],dx)\:|\:\psi' \mbox{in weak sense exists in $L^2([0,1],dx)$ and $\psi(1) = \chi\psi(0)$} \}\:. $$
(It is possible to prove that with the said definition of $D_\chi$ the definition is consistent: $\psi$ is continuous so that $\psi(0)$ and $\psi(1)$ makes sense.)
Next the self-adjoint extension of $P$ over $D_\chi(P)$ is again $-i\hbar \frac{d}{dx}$ where the derivative is interpreted in weak sense.
The simplest case is $\chi=1$ and you have the standard momentum operator with periodic boundary conditions, that is self-adjoint. The other self-adjoint extensions are trivial changes of this definition. I do not know the physical meaning of these different choices (if any): the theory is too elementary at this stage to imagine some physical interpretation. Maybe with an improved model a physical interpretation arises.
- What is the spectrum and eigenvectors of the momentum operator and its extensions? How do they differ from each other? Is there such a thing as a momentum representation in this setting? If not, why not?
You can easily compute the spectrum which is a pure-point spectrum and the eigenvectors are shifted exponentials. If $\chi = e^{i \alpha}$ where $\alpha \in \mathbb R$, and we denote by $P_\alpha$ the associated self-adjoint extension of $P$ a set of eigenvectors is
$$\psi^{(\alpha)}_n(x) = e^{i(\alpha + 2\pi n)x}$$
with eigenvalues
$$p^{(\alpha)}_n := \hbar(\alpha + 2\pi n)\quad n \in \mathbb Z\::$$
The set of the $\psi^{(\alpha)}_n$ is a Hilbert basis because it is connected with the standard basis of exponentials by means of the unitary operator $(U_\alpha \psi)(x) = e^{i\alpha x} \psi(x)$. Essentially Nelson's theorem and the spectral decomposition theorem prove that $P_\alpha$ has pure point spectrum made of the reals $p_n^{(\alpha)}$.
So a momentum representation exists as you can immediately prove.
- What is the relationship between the momentum operator (and its possible extensions) and the hamiltonian?
As you know, if you start from $H := -\hbar \frac{d^2}{dx^2}$ on $D(H):= \{\psi \in C^2([0,1]) \:|\: \psi(0)=\psi(1) =0\}$ (I assume $2m=1$)
this is essentially self-adjoint, though the corresponding momentum operator with domain (1) is not. (The proof immediately arises from Nelson's theorem since $H$ is symmetric and admits a Hilbert basis of eigenfunctions.)
However there are also different candidates for the Hamiltonian operator arising by taking the second power of each self-adjoint extension of $P$ with domain (1). The spectrum is made of the second powers the elements of the spectrum of the corresponding seof-adjoint extension $\hbar^2(\alpha + 2\pi n)^2$
Do they commute? Do they share a basis?
Momentum and associated Hamiltonian commute and a common basis is that written above for the momentum.
Different self-adjoint extensions and different Hamiltonians do not commute as you easily prove by direct inspection.
- Do these problems have counterparts or explanations in classical mechanics? (nudge, nudge)
I do not know
And, more importantly: what are good, complete and readable references where one can go and get more information about this?
I do not know, many results are spread in the literature. It is difficult to collect all them. A good reference is Reed and Simon's textbook: Vol I and II.
ADDENDUM. A technical point deserves a little discussion. Sometimes when introducing selfadjointness domains as above in the space $L^2(I)$, where $I\subset \mathbb{R}$ is a bounded interval, the functions $\psi$ are required to be absolutely continuous. This requirement is actually included in the condition that the weak derivative $\psi'$ exists and is included in $L^1$ (or $L^2$ since $I$ is bounded). In fact, a measurable function $\psi:I \to \mathbb{C}$ is absolutely continuous if and only if it admits weak derivative in $L^1(I)$. In this case, as the function is absolutely continuous, its derivative exists almost everywhere and coincides with $\psi'$.