You are saying "if one of the observers secretly sets up a measuring device near one of the slits without revealing it to anyone else, he'll know which slit the photon went through and thus not observe the interference pattern that all his colleagues still observe.", but this is not correct. In reality, if anyone alters the boundary conditions, that is, puts a detector on one of the slits, the interference pattern disappears, because the photon's distribution on the screen will become random.
It is very important to understand what happens to a photon when it interacts with the detector. One of the things that can happen to the photon is to be absorbed. Think about it, if the photon gets absorbed at the detector, it cannot be absorbed on the screen. Each and every single photon can only be absorbed once. The other thing that can happen to the photon is to get inelastically/elastically scattered. In this case, the photon gives or gives not some of its energy to the interacting atom in the detector, and changes angle, and then lands on the screen. The ultimate answer to your question is the angle. It changes randomly.
But then why does the interference pattern disappear? Because of the interaction. The photon interacts with the detector. The boundary conditions change. The photon will be out of phase with the other photons. The scattering photons change angle randomly. The pattern disappears.
You are asking why does the pattern disappear for everybody? Because the detector changes the boundary conditions and that is observer independent. The photons will interact with the detector and that interaction is observer independent too.
You are saying the pattern is in the observer, and not in the particle. In reality, the pattern is in (caused by) the whole QM system, including the particle, the wave, the slits.