My question is related to this question.
All is fine and dandy when we write down the formula $$f(A) = \sum f(a_i)|a_i \rangle \langle a_i |,$$
for $A$, let's say, self adjoint, which I suppose is a mnemonic to some extent saying "for $f(A)$, consider the corresponding function on $\mathbb{C}$, $f$, and put this evaluated at the eigenvalues of $A$ as the factor before each corresponding projector".
No problem. But what if I see an expression where I have something like $f(A,B)$, where $A,B$ are each self-adjoint? I can imagine two cases:
(1) $A,B$ commute, so that, using the simultaneous eigenbasis, I can still make sense of $\sum f(a_i,b_i)|a_i \rangle \langle a_i | $.
(2) $A,B$ do not commute. I am stumped as to what happens here -- do we simply say that $f(A,B)$ is not well-defined? I am not there yet but I have heard such contructions come up in the theory of scattering.
My question is what are the correct definitions in cases (1) and (2)?
Edit: I suppose depending on the structure of $f$, the argument may or may not be self-adjoint (e.g. if $f(A,B) = A-aB, a \in \mathbb{R}$ then we can take the eigenbasis of $C := A-aB$. But if $a \in \mathbb{C}$ then there is no guarantee in general that such a basis exists. I suppose such a case is a function of an operator which is not self-adjoint so perhaps we needn't worry (since such a function might not be well-defined in the first place).