How to deal with functions of multiple (self-adjoint) operators? My question is related to this question.
All is fine and dandy when we write down the formula
$$f(A) = \sum f(a_i)|a_i \rangle \langle a_i |,$$
for $A$, let's say, self adjoint, which I suppose is a mnemonic to some extent saying "for $f(A)$, consider the corresponding function on $\mathbb{C}$, $f$, and put this evaluated at the eigenvalues of $A$ as the factor before each corresponding projector".
No problem. But what if I see an expression where I have something like $f(A,B)$, where $A,B$ are each self-adjoint? I can imagine two cases:
(1) $A,B$ commute, so that, using the simultaneous eigenbasis, I can still make sense of
$\sum f(a_i,b_i)|a_i \rangle \langle a_i | $.
(2) $A,B$ do not commute. I am stumped as to what happens here -- do we simply say that $f(A,B)$ is not well-defined? I am not there yet but I have heard such contructions come up in the theory of scattering.
My question is what are the correct definitions in cases (1) and (2)?
Edit: I suppose depending on the structure of $f$, the argument may or may not be self-adjoint (e.g. if $f(A,B) = A-aB, a \in \mathbb{R}$ then we can take the eigenbasis of $C := A-aB$. But if $a \in \mathbb{C}$ then there is no guarantee in general that such a basis exists. I suppose such a case is a function of an operator which is not self-adjoint so perhaps we needn't worry (since such a function might not be well-defined in the first place).
 A: In short $f(A, B)$ is not defined for $A$ and $B$ operators.
By the notation $f(A, B)$ I'm imagining a polynomial function $f:\mathbb{C}^2 \to \mathbb{C}$. Something like
$$
f(x, y) = xy = yx
$$
Obviously this makes sense for scalars but if we "plug in" operators instead of scalars for the function arguments we then get something like
$$
f(A, B) \stackrel{?}{=}AB \stackrel{?}{=} BA
$$
Basically the notation $f(A,B)$ doesn't give us enough information to know whether $f(A, B) = AB$ or $f(A, B)=BA$.
That said, if the function $f$ was originally defined on some sort of non-commutative ring or something then that means it already had some ordering associated on it so then $f(A, B)$ would be defined.
The above addresses case (2) when $[A, B]\not=0$. As for case (1) if $[A, B] =0$ then sure, $f(A, B)$ is in general defined because it doesn't matter if it was $f(x, y) =xy$ or $yx$. But nonetheless, if you're dealing with operators, it's probably best practice to avoid notation like $f(A, B)$ unless you're very clear about what you mean by it.
A: This illustrates the point in terms familiar to most physicists. Indeed, your (1) makes sense, as you anticipated. For example, for $A=\hat x$ and $B=g(\hat x)$, you of course have $$
f(A,B)= \int\!\!dx~ |x\rangle f(x,g(x))~\langle x|~~.
$$
In your case (2), take the emblematic pair of noncommuting operators, $A=\hat x$ and $B=\hat p$. It's been proven dozens of times on this site that
$$
\hat x= \int\!\!dx~ |x\rangle  x\langle x|,\\
\hat p= \int\!\!dp~ |p\rangle  p\langle p| \\ 
=\int\!\!dpdxdx'~|x\rangle\langle x |p\rangle  p\langle p|x'\rangle \langle x'|
\\  =\int\!\!dx~ |x\rangle (-i\hbar \partial_x)\langle x| .
$$
The eigenvectors of $\hat x$ do not coincide with those of $\hat p$, but the latter may be "rotated" to the former, and, at the end of the day, yield the 2nd expression in terms of the former (eigenvectors), without  a visible connection to its set of eigenvectors.
You now readily get, e.g.,
$$
A^5 B^2= \hat x ^5 \hat p ^2=-\hbar^2 \int\!\!dx~ |x\rangle x^5   \partial_x^2\langle x| ,
$$
etc, unambiguous, in standard practice.
Fittingly, this is unequal to
$$B A^5 B= -\hbar^2 \int\!\!dx~ |x\rangle \partial_x x^5   \partial_x\langle x|,$$ or
$$B A B A^4=-\hbar^2 \int\!\!dx~ |x\rangle \partial_x x\partial_x x^4   \langle x|,$$ etc, in associative but non-commutative Heaviside calculus, as you may trivially verify  through these maps.
Takeaway: You cannot talk about the eigenvalues of A and B in the same breath, but you may still use the  eigenvectors of either to define a noncommutative function of them provided you have specified an ordering prescription for them, i.e. you define $f(x,-i\hbar\partial_x)$ here. You will not discover a quantization map this way: you will dictate it.

*

*Due diligence: Convince yourself this works for any polynomial of A and B in any ordering, so any noncommutative polynomial, hence any noncommutative Taylor series for the $f(A,B)$ with any ordering of A,B you chose and specify. Exponentials of non commuting variables occur routinely, while δ functions are but integrals of exponentials; the coefficients in the exponent need not be real.

