2
$\begingroup$

I do have a dataset containing nearly a decade of particulate matter measurements. At the lower precision limit of the measuring instrument, there are values given like < 8. For processing however, a defined value is needed.

What is the best possible way to deal with this?

  • Replacing it as missing values?
  • Replacing it by 8 or even 7 (< 8)?
  • Assuming 0?
  • Assuming the mean value between something like 0 and 8; e. g. 4?

Or is there a better solution, maybe considering the natural background value?

$\endgroup$

closed as off-topic by Wolpertinger, heather, user36790, Jon Custer, ACuriousMind Oct 3 '16 at 19:42

  • This question does not appear to be about physics within the scope defined in the help center.
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Draw a histogram to show how the number of readings in an interval varies with the mid-interval value. Does the low value tail of the histogram before $<8$ show a definite trend which can be extrapolated into the $<8$ region? $\endgroup$ – Farcher Oct 3 '16 at 10:12
  • 3
    $\begingroup$ stats.SE might be a good (better?) place to look for an answer to this question. $\endgroup$ – Ruslan Oct 3 '16 at 11:33
  • 2
    $\begingroup$ I'm voting to close this question as off-topic because this belongs on stats.SE, or maybe math.SE. $\endgroup$ – heather Oct 3 '16 at 11:38
  • 2
    $\begingroup$ I'm voting to close this question as off-topic because it has no conceptual physics query. $\endgroup$ – user36790 Oct 3 '16 at 11:51
2
$\begingroup$

It's worth thinking about why data are reported in that way.

When you fit a model to data, you should take account of the uncertainty. I expect that the uncertainty in your measurement is greater than 8 "units", which is why "<8" is even a thing - you obviously don't want to report negative numbers of particles (that would be unphysical), and when the number gets below that limit it seems to me that bias corrections might from time to time lead to negative numbers.

Clearly, a "very low number" should not be ignored; absent further information, you could use 4 as the number (being the unbiased estimate of the interval 0 - 8), but it is worth understanding more about the noise on the data and the corrections that are applied, before taking a decision.

This is one of those cases where there is not a single "correct" way to deal with numbers.

An analogy from another field: in CT imaging, individual detector pixels have an offset; this needs to be subtracted from the observed signal. When the observed signal is very low, the subtraction may result in a negative number; but since CT reconstruction involves the taking of the logarithm of the number, and the log of a negative number would really mess up the calculations, this creates a real problem. The solution is to use an iterative reconstruction ("data fitting") method that models the noise as part of the algorithm - this is how one avoids the negative-log issue, and it allows proper weighting of all the data points available.

$\endgroup$
  • $\begingroup$ The CT example is an interesting one. I've always thought that negative values are clipped to zero in measurements of light and similar signals. $\endgroup$ – Ruslan Oct 3 '16 at 13:24
  • $\begingroup$ @Ruslan in the case of CT, clipping to zero doesn't help (can't take log of negative number). Using a very small positive value in order to permit low dose CT imaging ends up giving a bias in the image. This is a problem that has limited low dose CT for such applications as PET attenuation correction, although recent work by Kinahan's group at UW in collaboration with GE (and possibly others) has found ways around this. $\endgroup$ – Floris Oct 3 '16 at 13:29
1
$\begingroup$

For simplicity, replace the situation by this scenario:

You have a scale which has markings starting from 10 to some number (as you didn't specify your upper limit). You measure 100s of different lengths and note down the readings. Now your readings have range from <10 to the upper limit. You are supposed to do calculations. What would you approximate <10 as? What if your lower limit was 0.01? So, whether you can approximate it to be zero, or variance, or mean, will give you some error, the amount of which depends on the values of your readings as a whole.

According to me, the calculations should be done (maybe taking a subset of uniformly taken data?) considering the values as missing, as zero, as 8, 7 and as mean. Then compare the results and see what is the difference between the answers.

You have data. That means you have to do some analysis, to see what fits your case the best. I earnestly hope someone experienced can give a better and direct solution.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.