# How to predict how much data to collect

The same question on CrossValidated

Apologies if I'm being a bit vague in what follows, I've been asked to keep certain aspects of the experiment confidential for the time being.

An analogous experiment would be like trying to 'see' the ebb-and-flow of the tide (0.5 day period) by locating a photon detector at the bottom of the ocean (though of course this wouldn't work and is silly, but the principle is at least quite similar.) Hope that clarifies it a bit, let me know if not.

I'm currently in the planning stages of this experiment that I am hoping will detect a 0.155% signal variation (relative magnitude) within a resonable time frame (less than 6 months ideally.) I've calculated the rate of (usable) data will be around 68 events per day, though it should be stressed this is a random variable. Now I'm trying to work out - how many days will I need to run the detector for to see the variation with a confidence level of 3σ?

Some other details that may (or may not) be relevant include: the variation in the signal is expected to be sinusoidal with a period of 0.5 days. For this reason I reduced my useful event rate to 34 (Ie half) as clearly there is no variation to see when the sinusoidal signal is at or close to the mean value.

I've been googling for a method to predict the size of a data set necessary to see such a small signal variation but have come up with nothing. I would be extremely grateful for any hints / tips anyone could offer.

-
Do you know the phase of the sinusoidal variation (i.e., then the maxima and minima occur?) Are you planning to search for the signal by fitting to a sine function with a known phase but unknown amplitude? Are your events expected to be Poisson distributed? In any case, if you want more than just a factor-of-a-few sort of estimate, you may have to simulate your data and run it through the analysis algorithm you plan to use. –  Ted Bunn Jul 11 '11 at 21:27
I have two advices. First: Ask mathematician, especially focused on statistics, because your task is approximation of some function. Second advice is maybe from 19th century but if nothing else works you can daily calculate variations and other statistic parameters and decide when to stop. –  Crowley Jul 11 '11 at 21:51
First you talk about measuring a signal, then about counting events. Which is it? Also, you might get better answers on crossvalidated.SE. –  Colin K Jul 11 '11 at 23:41
This is a good question, but it definitely belongs on stats.SE –  Colin K Jul 11 '11 at 23:44
@Ted Bunn: Yes, in theory it would be possible for me to match the collected data (time-stamped detection events) with a known phase. In practice this may be a little more difficult as the phase is not quite constant and would need regular adjustments. Unfortunately the detector will be place in a somewhat hard-to-reach site so the practicalities of accessing it regularly may prevent me doing so. Regarding whether the events are Poisson distributed, I'm not overly familier with the science/stastistics of RVs, but having looked it up I get the feeling that the event rate here most probably is. –  qftme Jul 12 '11 at 9:16

I don't think there's any way you're going to do this in six months.

I'll give a calculation below, but first an order of magnitude estimate. If you've detected a total of $N_{\rm events}$ events, your measurement of a modulation will have an error of order $N_{\rm events}^{-1/2}$ -- -- these things always do! -- so the number of events required is going to go like $1/f^2$ where $f$ is the modulation level you're looking for. In your case, $f=0.00155$, corresponding to about 400,000 events, which will take decades at the given event rate.

Now for the details.

Let $N_{\rm events}$ be the total number of events in your data set. Suppose that you bin your data into $N$ bins by time of day. You're assuming that the signal is of the form $$s_j=A+B\cos(t_j),$$ where $t_j$ is the time of day corresponding to the $j$th bin, and times of day are measured from the time when the signal is at its maximum. (If you don't know when that is and are planning to fit for it, then that'll change things.) Here $A$ is the average number of vents, so $$A=N_{\rm events}/N,$$ and $$B=fA={fN_{\rm events}\over N},$$ where $f=0.00155$ is the modulation.

Assuming further that your data are equally distributed across all times of day, the errors in $s_j$ will all be approximately equal (because $f$ is small). In this case, the best estimator of $B$ is $$\hat B={2\over N}\sum_j s_j\cos(t_j).$$ We want to find the variance $\sigma_B^2$ of this estimator. The individual $s_j$ are all independent and have nearly equal variances $\sigma^2$, so $$\sigma_B^2={4\sigma^2\over N^2}\sum_j\cos^2(t_j).$$ Assuming that $N$ is large enough that that sum can be approximated by an integral, the sum comes out to $N/2$, so $$\sigma_B^2={2\over N}\sigma^2.$$ For Poisson distributed events, the variance is equal to the expected value: $\sigma^2=A=N_{\rm events}/N$. Therefore, $$\sigma_B^2={2N_{\rm events}\over N^2}.$$ The fractional uncertainty is $${\sigma_B\over B}={\sqrt{2N_{\rm events}}\over N}{N\over fN_{\rm events}}=\sqrt{2\over f^2N_{\rm events}}.$$ For a 3-sigma detection, you want this to be equal to 1/3, so $$N_{\rm events}={18\over f^2}=2.5\times 10^6.$$ (My initial guess was off by a factor of 18 -- $3^2$ because of the 3 sigma, and 2 because of the point you noted about data near the zeroes of the modulation not helping.) At 68 events per day, this works out to about 300 years. Sorry.

-
thank you so much for taking the time to answer, this is exactly what I was after, with more detail than I had expected. Thanks again. –  qftme Jul 12 '11 at 9:29
I was just getting my head around the details of what you wrote and I have a quick couple of questions, I hope you don't mind. 1/ In the order-of-magnitude estimate are you saying one can take $N_{\rm events}^{-1/2}=f$ and solve for $N_{\rm events}$? I can replicate your "decades" prediction in this way but - where you said "your measurement of a modulation will have an error of order $N_{\rm events}^{-1/2}$" shouldn't it have had a positive power? 2/ What was the motivation for you choice of $\hat B$? I'm afraid I just can't work out where this bit comes from.. –  qftme Jul 12 '11 at 23:21
The thing that goes as $N_{\rm events}^{-1/2}$ is the error in the measurement of $f$. As you have more and more events, the error gets smaller and smaller, but only like the square root. To get a detection at a certain number of sigma, you need that error to be a given multiple of $f$ (e.g., 1/3 for a 3-sigma measurement). So the order-of-magnitude relation is $f\sim N_{\rm events}^{-1/2}$. –  Ted Bunn Jul 13 '11 at 15:11
The formula I gave for $\hat B$ is the best possible estimator. That means two things: First, it is unbiased -- in many repetitions of the experiment, the average of $\hat B$ would equal the true value. Second, it has minimum variance -- i.e., the smallest possible error. How did I know that this was the minimum-variance estimator? To be honest, just experience. When you look for a sinusoidally-varying quantity, the best thing to do is to multiply your data by the same sinusoidal function and integrate (or add up for discrete data). (More in next comment.) –  Ted Bunn Jul 13 '11 at 15:14
The $2/N$ in the formula for $\hat B$ comes from the fact that the "typical" term in this sum contains a contribution that averages out to $B/2$. So when you add up all $N$ terms, you get $(N/2)B$ on average. The reason that the typical term in the sum averages to $B/2$ is because each term in the sum gets a contribution $B\cos^2 t_j$ (one factor of cos coming from the original multiplication, and one coming from the "extra" cosine term in the definition of $\hat B$). Over a complete cycle, $\cos^2$ averages out to $1/2$. –  Ted Bunn Jul 13 '11 at 15:20

Back of the envelope calculation. (I'm rushed, hope I got this right.)

Probability questions like this are best done using probabilities, so first let's convert your estimate to a probability $p$:
Your signal variation is 0.00155 so:
$$1-2p = 0.00155$$ So $p = 0.499225$ and $1-p = 0.500775$. The standard deviation is
$$\sigma = \sqrt{p(1-p)/N} \approx \sqrt{1/(2N)}.$$

You want the standard deviation to be 1/3rd of the difference between 0.5 and $p$ so we solve for N:
$$(0.500775-0.5)/3 = \sqrt{1/(2N)}$$
to get $N= 7.5\times 10^6$.

At 68 events per day (actually it will be less because of the sine wave), this amounts to 21 thousand days.

-