Such objects are used all the time. The mathematics is done in terms of quantum fields, which to some extent conceals what's going on.
For example, your "(frequency-domain) coherence" is a correlation coefficient, which is normalized, whereas Physicists typically work in terms of correlation functions, which typically are not, but they largely amount to the same thing. Your observables $X(f_1)$, etc., are constructed as functions of frequency, however this is a singular object in quantum field theory. In quantum field theory, we instead construct observables $\phi(F_1)$, etc., as functionals of test functions $F_1(x)$, etc. One singular choice would be $F_1(x)=\exp(if_1\cdot x)$, which makes $\phi(F_1)$ essentially the same object as your $X(f_1)$; it's singular, however, because $F_1(x)$ is not square integrable.
Another choice of singular test function is, of course, $\delta(x-y)$, which gives the value of the field at a point, which we might write in your terms as something like $X(y)$. For a quantum field, this is also a rather singular object.
In fact, when you say $X(f_1)$, what you really mean is $\int X(f) {\mathrm d}f$, over some small range of frequencies, and in the mathematical and experimental details this has to be taken into account. Making everything precise requires that we know what the frequency ranges of each of the measurements is, which an experimenter either must characterize or must read off from a manufacturer's data sheets. In even more detail, we will have to construct a weight function, saying that frequencies near $f_1$ are still registered by the measurement device, but not as much as near $f_1$. We may well take the weight function, as a first approximation, to be Gaussian. This corresponds to taking the test function $F_1(x)$ to be that Gaussian. Signal analysis usually calls the test function a window function. Test or Window functions can be difficult to become familiar with, but I believe it's well worth getting there.
In these terms, your $C_{xyz}$ is a particular choice of (normalized) 3-point function. The choice of $f_1+f_2$ for the third frequency is of course not necessary, we can consider 3-point correlations between any three frequencies $f_1,f_2,f_3$ (and their vicinities). In quantum field theory, we would represent the 3-point correlation function, in the vacuum state, as $\left<0\right|\phi(F_1)\phi(F_2)\phi(F_3)\left|0\right>$. Replace the vacuum vector by some other state vector, if you like.
In the particular case when quantum field observables are mutually commutative, it can be understood to generate probability measures that correspond to an equivalent description in terms of probability measures over classical random variables, and hence quite precisely to a stochastic signal analysis. When quantum field observables do not commute, everything gets lots more complicated, but a remnant of the signal processing point of view can be maintained.
There is a mathematics of random fields that is used in cosmology because it is generally not necessary to worry about measurement incompatibility in that context. Mathematicians generally present signal analysis in Hilbert space terms unless they are writing for an engineering audience.
