How to predict how much data to collect

Question

The same question on CrossValidated

Apologies if I'm being a bit vague in what follows, I've been asked to keep certain aspects of the experiment confidential for the time being.

An analogous experiment would be like trying to 'see' the ebb-and-flow of the tide (0.5 day period) by locating a photon detector at the bottom of the ocean (though of course this wouldn't work and is silly, but the principle is at least quite similar.) Hope that clarifies it a bit, let me know if not.

I'm currently in the planning stages of this experiment that I am hoping will detect a 0.155% signal variation (relative magnitude) within a resonable time frame (less than 6 months ideally.) I've calculated the rate of (usable) data will be around 68 events per day, though it should be stressed this is a random variable. Now I'm trying to work out - how many days will I need to run the detector for to see the variation with a confidence level of 3σ?

Some other details that may (or may not) be relevant include: the variation in the signal is expected to be sinusoidal with a period of 0.5 days. For this reason I reduced my useful event rate to 34 (Ie half) as clearly there is no variation to see when the sinusoidal signal is at or close to the mean value.

I've been googling for a method to predict the size of a data set necessary to see such a small signal variation but have come up with nothing. I would be extremely grateful for any hints / tips anyone could offer.

Answer 1

I don't think there's any way you're going to do this in six months.

I'll give a calculation below, but first an order of magnitude estimate. If you've detected a total of $N_{\rm events}$ events, your measurement of a modulation will have an error of order $N_{\rm events}^{-1/2}$ -- -- these things always do! -- so the number of events required is going to go like $1/f^2$ where $f$ is the modulation level you're looking for. In your case, $f=0.00155$, corresponding to about 400,000 events, which will take decades at the given event rate.

Now for the details.

Let $N_{\rm events}$ be the total number of events in your data set. Suppose that you bin your data into $N$ bins by time of day. You're assuming that the signal is of the form $$ s_j=A+B\cos(t_j), $$ where $t_j$ is the time of day corresponding to the $j$th bin, and times of day are measured from the time when the signal is at its maximum. (If you don't know when that is and are planning to fit for it, then that'll change things.) Here $A$ is the average number of vents, so $$ A=N_{\rm events}/N, $$ and $$ B=fA={fN_{\rm events}\over N}, $$ where $f=0.00155$ is the modulation.

Assuming further that your data are equally distributed across all times of day, the errors in $s_j$ will all be approximately equal (because $f$ is small). In this case, the best estimator of $B$ is $$ \hat B={2\over N}\sum_j s_j\cos(t_j). $$ We want to find the variance $\sigma_B^2$ of this estimator. The individual $s_j$ are all independent and have nearly equal variances $\sigma^2$, so $$ \sigma_B^2={4\sigma^2\over N^2}\sum_j\cos^2(t_j). $$ Assuming that $N$ is large enough that that sum can be approximated by an integral, the sum comes out to $N/2$, so $$ \sigma_B^2={2\over N}\sigma^2. $$ For Poisson distributed events, the variance is equal to the expected value: $\sigma^2=A=N_{\rm events}/N$. Therefore, $$ \sigma_B^2={2N_{\rm events}\over N^2}. $$ The fractional uncertainty is $$ {\sigma_B\over B}={\sqrt{2N_{\rm events}}\over N}{N\over fN_{\rm events}}=\sqrt{2\over f^2N_{\rm events}}. $$ For a 3-sigma detection, you want this to be equal to 1/3, so $$ N_{\rm events}={18\over f^2}=2.5\times 10^6. $$ (My initial guess was off by a factor of 18 -- $3^2$ because of the 3 sigma, and 2 because of the point you noted about data near the zeroes of the modulation not helping.) At 68 events per day, this works out to about 300 years. Sorry.

Answer 2

Back of the envelope calculation. (I'm rushed, hope I got this right.)

Probability questions like this are best done using probabilities, so first let's convert your estimate to a probability $p$:
Your signal variation is 0.00155 so:
$$1-2p = 0.00155$$ So $p = 0.499225$ and $1-p = 0.500775$. The standard deviation is
$$\sigma = \sqrt{p(1-p)/N} \approx \sqrt{1/(2N)}.$$

You want the standard deviation to be 1/3rd of the difference between 0.5 and $p$ so we solve for N:
$$(0.500775-0.5)/3 = \sqrt{1/(2N)}$$
to get $N= 7.5\times 10^6$.

At 68 events per day (actually it will be less because of the sine wave), this amounts to 21 thousand days.