Two real gravitons can scatter by exchanging a virtual graviton, when one quantizes small perturbations in General Relativity. So your “fact” is not a fact.
See section 4.3 of this paper for the Feynman diagrams and a calculation of the scattering amplitude. (Note: I think this calculation was first done by deWitt, but his landmark 1967 paper is behind a paywall. It is now common enough to just be an exercise for a masters thesis!)
The reason this can happen for gravitons and not photons is that gravity is nonlinear while electromagnetism is linear. In terms of Feynman vertices, there are vertices where three graviton lines or four graviton lines can interact, but no vertices where just photon lines can interact. The only vertex in QED is one where two charged-particle lines and one photon line meet.
QCD is also nonlinear and has three- and four-gluon vertices, similar to those for gravitons. But color confinement means that we don’t observe real gluons.
Of course, we don’t observe real gravitons either. But this is more for technological reasons (much too difficult, because gravity is so weak) than for theoretical ones. It’s true that the quantized GR that predicts gravitons is not a consistent theory, but string theory also predicts gravitons. I don’t know whether graviton-graviton scattering via virtual gravitons has been calculated in string theory.
Finally, string theory may not be correct, but most physicists assume that gravity must be some kind of quantum field just like all other fundamental fields. So everyone expects that whatever theory of quantum gravity emerges as the winner is going to have gravitons, is going to be nonlinear, and is going to reduce to quantized GR at low energies.