Check the image below, from this excellent explanation of the quantum delayed eraser. My question, and please forgive me if it is a naive one, is that first the photon reaches detector A, which tells us whether we have an interference pattern or not. With this information in hand, and because the second detection to know the which-path information is delayed for minutes or even millions of years, won't we have enough time to just remove the splitters and always send the entangled particle to detectors 2 and 3, effectively revealing the which-path information and causing a contradiction?
My point is that if I first look to find an interference pattern on detector A and then wait for a couple of millions years so I have time to effectively remove the splitters before going back to look again at detector A, will I find that my first observation has mysterious changed to the double line pattern, in other words, the future has just changed the past because what I first saw with my own eyes was an interference pattern that has now turned to a double line pattern?
Has anyone tried this experiment?