Delayed Choice Quantum Eraser - what is erased? I fail to see why everybody thinks that something is erased in the famous Delayed Choice Quantum Eraser experiment (see https://arxiv.org/pdf/quant-ph/9903047.pdf and https://en.wikipedia.org/wiki/Delayed-choice_quantum_eraser).
According the paper, the interference patterns that arrive at D1 and D2 are shifted by pi. That is, if you add the interference patterns together, the pattern is the same as the pattern for detectors D3 and D4. Therefore, if you look only at D0, then no interference pattern is visible, not for the photons whose peer arrive at D3 and D4, but also not for the photons whose peer arrive at D1 or D2. This is different from the normal double slit experiment.
Therefore, isn't it equally valid to say that the interference is always there, also for the photons that arrive at D3 or D4? We just don't detect it in that case, because we only look at only one of the possible paths and don't combine them together. It is the beamsplitter BSc for D1 and D2 which makes the interference pattern visible. That is, no information is erased, and there is no need to transfer information from D1/2/3/4 to D0.
Edit: expressed in a different way: if I would remove the D3/D4 mirrors, and let all photons go through to D1/D2, it would clearly show that there is an interference pattern in D0 for all photons. Setting back the D3/D4 mirrors, redirecting 50% of the photons, would change nothing at the D0 side; the photons would still hit at exactly the same place at D0 as before. That is, the interference pattern would still be there, we just would not have enough information to detect it afterwards.
What is wrong with this interpretation of the results? And if this interpretation can be valid, why is this such a controversial experiment? 

 A: If one takes what you've said in the comments at face value, particularly in this exchange,

If a physicist has observed which of D3 and D4 detected a photon, isn't it impossible to recover the interference? (Unless you believe in the many-world interpretation, and do some kind of inter-world interferometry on the physicist.) – Peter Shor
Exactly, that's my point. You cannot recover the interference. But that you cannot recover it does not mean that it is erased. You simply don't have enough information to recover it. – fishinear

it seems that you are completely OK with suddenly re-defining the boundaries of the experiment to a point where you can arbitrarily decide to (i) treat everything in the experiment, including the detectors, computers, and any humans who have interacted with those computers, as fully quantum system undergoing unitary Schrödinger evolution, and to which classical physics does not apply; and (ii) to perform arbitrarily-complex superposition-basis measurements on those quantum systems, regardless of the possibility that they might contain human beings.
This is a rather radical posture, though it is not unthinkable, particularly in many-worldsian circles - and it is a valid interpretation of the experiment, consistent with the rules of QM as we know it. Within this posture, it is indeed true that interference is never truly gone, and that the which-way information that destroys that interference can be erased at any point by performing the procedure above.
However, it is not a widely-shared interpretation, and it is explicitly dependent on which interpretation of QM you choose to follow; as such, it cannot be "wrong". But also, the set of professional physicists who would describe that posture as being "right" is rather limited.
(If that  middle paragraph doesn't accurately describe what you're OK with countenancing in the description of this experiment and QM in general, then that's a sign that your question still isn't clear enough, and you need to go back to the edit board.)
A: I now managed to find an answer to another question that I think satisfies my question as well. A quote:

The exact same logic applies to the delayed choice quantum eraser experiment, except there's an extra value involved and you're looking for interference patterns instead of passing bell tests. No consciousness. No retrocausality. Just "did we get and use the distinguishing information needed to group the lack-of-interference pattern into two complementary interference patterns"

Here is the answer: https://physics.stackexchange.com/a/214926/101743
To summarise my understanding: the x-position on D0 is really a measurement of the initial phase difference between the two paths. This phase difference is also what is filtered on for the D1/D2 detectors, which gives rise to the interference pattern. Therefore this, phase difference is what I actually meant in the question when saying "the interference is still there". 
My understanding is now that the phase difference and the which-way information are independent properties of the wave-function, which are both there until the wave-function collapses. Neither of them are "erased" in the experiment. When the photon arrives at D3/D4 only the which-way information is used, when it arrives at D1/D2 only the phase difference is used. 
