A bit of background helps frame this question. The question itself is in the last sentence.
For his PhD thesis, Richard Feynman and his thesis adviser John Archibald Wheeler devised an astonishingly strange approach to explaining electron-electron interactions without using a field. Their solution was to take the everyday retarded wave solution to Maxwell's equations and mix it 50/50 with the advanced (backwards in time) solution that up until then had always been discarded as "obviously" violating temporal causality. In a series of papers they showed that this was not the case, and that the recoil of an electron when it emits a photon could self-consistently be explained as the result of an advanced photon traveling backwards in time and impacting the electron at the same instant in which the electron emits a forward-in-time photon.
While Feynman's thesis ideas deeply influenced his subsequent development of QED, e.g. in QED's backwards-in-time interpretation of antimatter electrons (positrons). Feynman in a letter to Wheeler later famously retracted (well, famously for some of us) the specific idea of paired photons traveling forwards and backwards in time. Feynman's specific reason for abandoning his thesis premise was vacuum polarization, which cannot be explained by direct electron-to-electron interactions. (Vacuum polarization is easily accommodated by QED, however.)
Feynman's abandonment of the original Feynman/Wheeler retarded/advanced photon scheme has always troubled me. The reason is this: If their original idea was completely invalid, the probability from an information correlation perspective of the idea leading to accurate predictions of how physics operates should have been vanishingly small. Instead, their oddly arbitrary 50/50 mix of real waves and hypothesized backwards-in-time waves almost works, all the way down to the minute length scales at which vacuum polarization becomes significant. One analogy is that the Feynman/Wheeler scheme behaves like a slightly broken mathematical symmetry, one that correctly describes reality over almost the entire range of phenomena to which it applies, but then breaks down at one of the extrema of its range.
My question, finally, is this: Does there exist a clear conceptual explanation, perhaps in the QED description of vacuum polarization for example, of why the Feynman/Wheeler retarded/advanced model of paired photons traveling two directions in time provides an accurate model of reality overall, despite its being incorrect when applied at very short distances?
If I've understood @RonMaimon correctly -- and I certainly still do not fully understand the S-matrix part of his answer -- his central answer to my question is both simple and highly satisfying: Feynman did not abandon the backward-forward scheme at all, but instead abandoned the experimentally incorrect idea that an electron cannot interact with itself. So, his objection to Wheeler could perhaps be paraphrased in a more upbeat form into something more like this: "Vacuum polarization shows that the electron does indeed interact with itself, so I was wrong about that. But your whole backwards-and-forwards in time idea works very well indeed -- I got a Nobel Prize for it -- so thank for pointing me in that direction!"
Answer to Ron, and my thanks.