Delayed choice experiment clarification (concrete setup) I know that clarifications about delayed choice experiment was asked million times, and I understand the idea, but I was not able to find the discussion of this particular situation anywhere, though I tried hard. This setup is described in Brian Greene's Fabric of the Cosmos book (note my question is about modification of this setup, not about it exactly as described in book). Here is a picture of setup from the book

So photon goes from source to the beam splitter, then there are two down-converters that split photon into two, each of which goes in different directions: one follows the previous path (that one is named "signal") and the other goes into different direction to the detector ("idler" photon). Two signal photons then go to detector. Without down-converters we see interference pattern in detector - that seems clear. With down-converters it is claimed that we don't see interference pattern because detecting idler photons reveals information about which path photon has "chosen": if photon detected in top detector - photon has chosen top path, if not - then bottom one. So interference pattern disappears (so is claimed in the book).
Then book follows to description of modification of this setup to erase which path information - this part is clear for me and it's not what I'm interested about. What is not clear is:
What if we remove idle photon detectors from the setup, but leave down-converters in place? Then it seems "which path" information is not revealed and interference pattern should appear again? But that does not make much sense, because we can decide to put or not a detector at much later time (after corresponding singal photon hit the screen). So depending on if we see pattern or not we should know what happened in the future, which of course cannot happen. Is that assumption about removing idler detectors leads to interference pattern incorrect? If so, why? What will happen if we switch idler detectors on and off by our choice? I suspect this has something to do with the fact we now have two photos instead of one (two photons from down-converters). Maybe those photons have different phases and combined they do not form any pattern?
 A: This is a very appealing way to think:

What if we remove idle photon detectors from the setup, but leave down-converters in place? Then it seems "which path" information is not revealed and interference pattern should appear again? 

However, this is not the case: what matters is that the information is there to be detected, even in principle. It doesn't matter whether you put detectors in or not, whether they are turned on or whether there is a computer or a human being there to record the measurements: as long as the information is there to be recorded, this which-way information will wash out the interference pattern.
Now, this is a hard pill to swallow, and much ink has been spilled trying to make sense of it, so I won't try to sugar-coat it here. It is weird and counter-intuitive, but the way out you're hoping for here is simply not there. 
I should mention that there are some ways to restore the interference pattern even after the idler photons have been generated, known as quantum eraser experiments, and they work by completely and provably wiping out the which-way information carried by the idler photons. (In this specific case, you'd route both idlers to a beam splitter and put the detectors behind that. Globally, the interference will be gone, but it will come back if you post-select on each of the two outputs of the beam splitter, i.e. essentially by projecting on $|\mathrm{idler}_1⟩|0_2⟩ \pm |0_1⟩|\mathrm{idler}_2⟩$.) 
However, that's a different situation to what you're asking about - simply turning off the detection won't help you there. Once the idler is emitted, the signal photons are entangled with the idlers, and entangled particles typically don't produce this sort of interference as easily.
