Double-slit experiment: How do we know the particle effect comes from the nature of light rather than its interaction with the detector? In the double-slit experiment, we shine a light wave through two closely-spaced parallel slits at a screen, and observe an interference pattern on the screen. We then reduce the intensity of the light source until we can observe single energy quanta, like Geiger counter clicks, which we call photons, and we observe the same interference pattern in the distribution of where the quanta arrive. We infer that light behaves in some ways like a wave, because it creates an interference pattern, and in some ways like a particle, because we can count the individual photons. It seems like a single photon can interfere with itself. Weird!

Question: How do we know the light travels in quanta? Could it not be the case that the light has a continuous wave nature, and only the interaction between the light wave and the source, and the light wave and the detector, is quantized? Separately from each other, of course, because of causality.
If this hypothesis was true, we should see that quanta would be deposited on the detector at the same average rate they were removed from the source, but there would be no correlation between the precise timing of each. In the usual experimental setup, there is no way to know this because the energy quanta taken from the source are not counted. Has this been experimentally tested?
 A: You are right that the the experiment is compatible with a classical description of light (an EM wave). There is a theory, Stochastic Electrodynamics, capable of reproducing many results of quantum mechanics, like black body radiation, specific heat of solids and, to some extent, the stability of atoms.
The way this is done is to assume that the so-called quantum fluctuations are actually a real EM field that permeates all space and can be experimentally measured as Casimir force.
You can take a look at this paper:
Stochastic Interpretation of Quantum Mechanics Assuming That Vacuum Fields Are Real
Emilio Santos, Foundations 2022, 2(2), 409-442.
https://www.mdpi.com/2673-9321/2/2/28
At paragraph 5, The Particle Behaviour of Light we read:
"Firstly, I point out that the absorption of light in the form of localized spots in a photographic plate or clicks in a photodetector are not valid arguments for the particle behaviour of radiation. In fact, the former is caused by the granular (atomic or molecular) nature of photografic plates. The latter derive from the fact that photocounters are manufactured so that they click when the radiation arriving during a detection time surpasses some threshold, which is compatible with light being continuous (waves)."
Hopefully, this answers your question.
A: 
If this hypothesis was true, we should see that quanta would be
deposited on the detector at the same average rate they were removed
from the source, but there would be no correlation between the precise
timing of each. In the usual experimental setup, there is no way to
know this because the energy quanta taken from the source are not
counted. Has this been experimentally tested?

A common application of this is a Compton Telescope, where the photon resulting from the interaction in the first detector shows up at the second detector just when you'd expect.
Your mistake is to expect nature to match the mathematical objects you wish to use to model it. This is a common misunderstanding due to the way we teach physics as mathematics on a whiteboard. Students must accept this to get a passing grade, but nature is under no obligation to conform.
The way physics contacts reality is through experiments. The math models the patterns the experiments reveal, often very effectively, but it is never "real".
"Stop telling God what to do." -Neils Bohr
