4
$\begingroup$

Disclaimer: I am aware this is somehow at the boundary between physics and statistics, but I have the impression that it is more likely that somebody doing/studying physics, rather than statistics, has stumbled in this problem, which in the end is very practical. Feel free to comment and/or suggest a better community if you disagree!


When measuring any physical parameter, uncertainty can be reduced by repeating the measurement many times and then taking the average value $\mu=(\sum_1^N x_i)/N$, or also by measuring for a longer time. Assuming the single measurement has a noise $\sigma_{x_i}=\sigma_0$, it is common knowledge that the standard deviation of the average of $N$ measurement should scale as $\sigma_\mu = \sigma_0/\sqrt{N}$ (or actually $\sigma_0/\sqrt{N-1}$)... so the more you measure, the more precise your estimate.

However, indeed this is only true if the $N$ measurements are uncorrelated, which is equivalent to stating that the measurement is affected only by white noise. When flicker noise is present (i.e., ultimately, always), the power spectral density of noise deviates from flat in the low-frequency limit, and at some point starts growing as $\propto 1/\omega$ (or even steeper but let us neglect that). As a result, this means that the value you measure "now" has some correlation with those you got "previously", and the rule above fails.

Question: How does the standard deviation scale with the number of measurements or with the integration time in the presence of flicker/pink noise?! Can you point me in the right direction and/or do you have a good textbook/reference discussing this? I am having a hard time finding one... statistics books even rarely discuss power spectral densities, not to mention specific cases like this one; in physics texts, it is uncommon that correlated noise is discussed in detail.

What I have concluded so far. It is quite obvious that in the presence of pink noise the precision of the estimate won't improve forever like $\approx 1/\sqrt{N}$ by simply increasing the number of measurements $N$. I have found vague comments about the existence of a "Flicker floor" but no real clear discussion about it. I see that the problem has a well-defined frequency scale, which is the frequency below which flicker noise takes over and dominates the spectral density, the "Flicker corner" $\omega_c$. So I would intuitively expect that measuring for a time that is longer or shorter than the time scale $1/\omega_c$ makes a qualitative difference.

One obvious approach to see what happens if I integrate my measurement for a time $T$ might be to consider the measurement window - say, $[-T/2,+T/2]$ - and use its Fourier transform (sinc function) to see which are the frequencies that I am sampling and calculate the total noise. With white noise, it works fine but when you turn to pink, the integral of the PSD clearly diverges. I suspect this has to do with the fact that even the average does not converge with flicker... but... not sure where to head to from here.

[Edit] I have found some interesting clues, "flicker floor" was the main search key that led me to the link below. I have not gone through the work in details but it seems to answer to at least some of my doubts

https://arxiv.org/abs/1407.7760v6

$\endgroup$
4
  • $\begingroup$ Very short comment: if you apply the PSD of the pink noise to use when "generating" samples, you'll start to see how the number of samples needs to be "weighted" related to the probability curve. $\endgroup$ – Carl Witthoft Jun 5 at 14:05
  • $\begingroup$ You may have to analyze a covariance matrix of all measurements, rather than their standard devuations (diagonal elements of this matrix). Also, it might be worthwhile to look at it as time series (using one of the available models: ARMA, ARIMA, etc.) Just to give you some keywords. $\endgroup$ – Roger Vadim Jun 5 at 14:27
  • $\begingroup$ Thanx Roger and Carl! I try and look further into the thing with these inputs. Indeed I tried once to compute the covariance matrix elements and proceed from there: I cant remeber how it ended exactly, I just recall I got lost but I might try again. $\endgroup$ – Ste Jun 5 at 16:49
  • $\begingroup$ @Ste there's a basic problem that the noise power is infinite if the 1/f power spectral density extends all the way down to zero. Nevertheless, if the wanted signal has a distinct spectrum (i.e. not simply a dc constant value) , there are methods to optimally estimate (filter) the signal. $\endgroup$ – Roger Wood Jun 11 at 4:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.