What does the absolute value of an operator mean? In the text, we are given a Hamiltonian for two bodies with a potential energy of interaction that depends on the magnitude of the distance between them as:
$$\hat{H}=\frac{\hat{\mathbf{p}}_{1}^{2}}{2 m_{1}}+\frac{\hat{\mathbf{p}}_{2}^{2}}{2 m_{2}}+V\left(\left|\hat{\mathbf{r}}_{1}-\hat{\mathbf{r}}_{2}\right|\right).$$
What does the absolute value of an operator mean and how does it operate and why?
 A: Consider any quantum mechanical observable $\hat O$ so that  $$\hat O|\psi⟩=o|\psi⟩$$ with eigenvalue $o$. Then for any function $f$ of the observable $\hat O$ we have, $$f(\hat O) \mid\psi\rangle = f(o) \mid\psi\rangle$$
The same applies for a function of the absolute value operator. So if we consider $V(\mid\hat{\mathbf{r}}_{1}-\hat{\mathbf{r}}_{2}|)$ and the two-particle state $\mid \psi(\mathbf{r_1}, \mathbf{r_2})\rangle$ then $$V(\mid\hat{\mathbf{r}}_{1}-\hat{\mathbf{r}}_{2}|)\mid \psi(\mathbf{r_1}, \mathbf{r_2})\rangle=V(\mid{\mathbf{r}}_{1}-{\mathbf{r}}_{2}\mid)\mid \psi(\mathbf{r_1}, \mathbf{r_2})\rangle$$ where the potential on the RHS is now a function of the eigenvalue $(\mid{\mathbf{r}}_{1}-{\mathbf{r}}_{2}\mid)$.
A: If an operator $A$ is normal$^\ddagger$ - that is, $[A^\dagger,A]=0$ - it admits a spectral decomposition of the form
$$A = \sum_i \lambda_i |\lambda_i\rangle\langle\lambda_i| $$
where the $\lambda_i$ are its (possibly generalized) eigenvalues, the $|\lambda_i\rangle$ its (possibly generalized) eigenvectors, and the sum may be replaced by an integral if the spectrum of $A$ is continuous.  In such cases, given a function $f:\mathbb C\rightarrow \mathbb C$, one may define the operator $\hat f(A)$ to be
$$\hat f(A):= \sum_i f(\lambda_i) |\lambda_i \rangle\langle \lambda_i|$$
In particular, this holds for the absolute value function.
It's possible to generalize this construction beyond normal operators by noting that for any$^{\ddagger\ddagger}$ $A$, the operator $A^\dagger A$ is self-adjoint (and therefore normal) and in fact positive, such that
$$A^\dagger A = \sum_i \Lambda_i |\Lambda_i\rangle\langle \Lambda_i| , \qquad \Lambda_i \in[0,\infty)$$
That being the case, we are able to apply the square-root function to $A^\dagger A$.  From here, we define $|A|$ to be
$$|A| := \sqrt{A^\dagger A} = \sum_i \sqrt{\Lambda_i} |\Lambda_i \rangle\langle \Lambda_i|$$
As an example of the latter general construction, consider the absolute value of the creation operator $a^\dagger$ from the quantum harmonic oscillator problem.  $[a,a^\dagger]\neq 0$ so it is not normal, but $ aa^\dagger = 1+ a^\dagger a = 1 + N$ where
$$N = \sum_{n=0}^\infty n | n\rangle\langle n|$$
is the number operator.  By definition then, we have that
$$|a^\dagger| := \sqrt{aa^\dagger} = \sum_{n=0}^\infty \sqrt{1+n}|n\rangle\langle n|$$
Note that $a^\dagger$ has no eigenvalues or eigenvectors, so one cannot define its absolute value spectrally. This definition of the absolute value of an operator is used in the polar decomposition of an operator.

$^\ddagger$ Familiar examples of normal operators are self-adjoint operators, for which $A^\dagger = A$, and unitary operators, for which $A^\dagger = A^{-1}$.
$^{\ddagger\ddagger}$The usual subtleties and caveats (domain issues, etc) apply if the operator is unbounded, but the moral of the story remains the same.
