Let $A:\mathcal D(A)\rightarrow \mathcal H$ be a densely-defined, symmetric operator on a Hilbert space $\mathcal H$. Consider the following definition for the Cayley transform:
$$\mathcal C(A) = (A-i\mathbb 1)(A+i\mathbb 1)^{-1}$$
First, a few preliminary notes. In the following, $\mathcal D$ denotes the domain of an operator and $\mathcal R$ denotes its range.
- $\mathcal D\big((A+i\mathbb 1)^{-1}\big) = \mathcal R(A+i\mathbb 1)$ and $\mathcal R\big((A+i\mathbb 1)^{-1}\big) = \mathcal D(A+i\mathbb 1) = \mathcal D(A)$. Consequently, we have that $\mathcal D\big(\mathcal C(A)\big) = \mathcal R(A+i\mathbb 1)$ and $\mathcal R\big(\mathcal C(A)\big) = \mathcal R(A-i\mathbb 1)$.
- $\forall \psi\in\mathcal D\big(\mathcal C(A)\big)$, $C(A)$ is an isometry, so $\Vert \mathcal C(A)\psi\Vert = \Vert\psi\Vert$.
- $\mathcal C(A)$ does not have $1$ as an eigenvalue, because $\mathcal C(A)\psi = \psi \iff (A-i\mathbb 1)\psi = (A+i\mathbb 1)\psi\iff\psi=0$. Therefore, letting $U=\mathcal C(A)$ we can define the inverse transformation
$$A=\mathcal C^{-1}(U) = i\big(\mathbb 1+U\big)\big(\mathbb 1-U\big)^{-1}$$
If $A'\supseteq A$ is a symmetric extension of $A$, then $\mathcal C(A')$ is an isometric extension of $\mathcal C(A)$. If $A'$ is a self-adjoint extension of $A$, then $\mathcal C(A')$ is a unitary extension of $\mathcal C(A)$. Conversely, if $\mathcal C(A')$ is a unitary extension of $\mathcal C(A)$, then $\mathcal A'$ is a self-adjoint extension of $A$.
Having established these facts, let $A_0$ be a densely-defined, symmetric operator and $\mathcal C(A_0)$ its Cayley transform, which as a reminder is an isometric map from $\mathcal R(A_0+i\mathbb 1)\rightarrow \mathcal R(A_0-i\mathbb 1)$. We seek some unitary extension $\mathcal C(A)\supseteq \mathcal C(A_0)$.
The key questions are (i) if such an extension can possibly exist, and if so, (ii) how much freedom we have in defining it. Being an extension, it must agree with $\mathcal C(A_0)$ on $\mathcal R(A_0 + i\mathbb 1)$; being unitary, it must be a bijection between $\mathcal R(A_0+i\mathbb 1)$ and $\mathcal R(A_0 - i \mathbb 1)$. Therefore our freedom (if any exists) lies how we define $\mathcal C(A)$ to act on $\mathcal R(A_0+i\mathbb 1)^\perp$. There are three possibilities.
If $\mathcal R(A_0+i\mathbb 1)^\perp =\mathcal R(A_0-i\mathbb 1)^\perp = 0$, then the only possible unitary extension $\mathcal C(A)\supseteq \mathcal C(A_0)$ is already determined and no freedom remains. In this case, the domain and range of $\mathcal C(A)$ are $\overline{\mathcal R(A_0+i\mathbb 1)} = \mathcal H$ and $\overline{\mathcal R(A_0-i\mathbb 1)} = \mathcal H$, where the line denotes topological closure. The corresponding $A$ is the unique self-adjoint extension of $A_0$.
If $\mathcal R(A_0+i\mathbb 1)^\perp$ and $\mathcal R(A_0-i\mathbb
1)^\perp$ are complex vector spaces of the same (non-zero) dimension, then we
can define $\mathcal C(A)$ to be any bijection between them. If
these spaces have finite dimension $n$, then the corresponding
unitary group has dimension $n^2$, so our possible unitary extensions are
parameterized by $n^2$ quantities.
If $\operatorname{dim}\big(\mathcal R(A_0+i\mathbb 1)^\perp\big) \neq \operatorname{dim}\big(\mathcal R(A_0+i\mathbb 1)^\perp\big)$, then it's not possible to define a unitary extension of $\mathcal C(A_0)$, so $A_0$ does not admit any self-adjoint extensions.
In light of this, we define the deficiency indices $d_{\pm} := \operatorname{dim}\big(\mathcal R(A_0\pm i\mathbb 1)^\perp\big) = \operatorname{dim}\big(\operatorname{ker}(A_0^*\mp i\mathbb 1)\big)$. Note that the deficiency indices are the dimensionalities of the eigenspaces of $A_0^*$ corresponding to eigenvalues $\pm i$.
If $d_+=d_-=0$, then there is only one self-adjoint extension of $A_0$, and $A_0$ is essentially self-adjoint. If $d_+=d_-\equiv d\neq 0$, then $A_0$ admits an infinity of self-adjoint extensions parameterized by $d^2$ quantities. If $d_+\neq d_-$, then $A_0$ admits no self-adjoint extensions at all.
Is $H^{**}=H$ a typo (should it be $H^*=H$ instead), or is there something else I'm missing here?
If $H$ is essentially self-adjoint, then its closure $H^{**}$ is self-adjoint, i.e. $(H^{**})^* = H^{**} \implies H^* = H^{**}$ where we've used the fact that $H^*$ is already closed since $H$ is densely-defined and symmetric. $H^{**}=H$ means that $H$ is closed, not that it is essentially self-adjoint, so I think that is indeed a typo and should read $H^{**}=H^*$.
Example: Consider this candidate for a momentum operator for a particle in a box, with $\mathcal H = L^2([0,2\pi])$.
$$\mathcal D(P_0):=\left\{\psi\in AC([0,2\pi])\ \big| \ \psi(0)=\psi(2\pi)=0\right\}$$
$$\big(P_0 \psi\big)(x)=-i\psi'(x)$$
Here $AC$ refers to the set of absolutely continuous functions. It's not difficult to show that $P_0$ is densely-defined and symmetric. However, it is not self-adjoint; the adjoint of this operator has domain
$$\mathcal D(P_0)=AC([0,2\pi])$$ with no boundary conditions at all.
Observe that
$$(P_0^*\pm i\mathbb 1)\psi = 0 \iff \psi(x) = a_{\pm} e^{\pm x}$$
for arbitrary $a_\pm\in\mathbb C$; as a result, $d_+=d_-=1$. $A_0$ is not essentially self-adjoint, but it admits an infinite family of self-adjoint extensions. The range of $(P_0\pm i\mathbb 1)$ must grow to be dense in $\mathcal H$, and we can do this by admitting more functions to the domain of $P_0$. Concretely, for each $\theta\in\mathbb R$ the following family of operators (each an extension of $P_0$) are all self-adjoint:
$$\mathcal D(P_\theta) := \left\{\psi\in AC([0,2\pi]) \ \big| \ \psi(0)=e^{i\theta}\psi(2\pi)\right\}$$
$$\big(P_\theta \psi\big)(x) = -i\psi'(x)$$
