I have some experimental data, and since the measurements are digital, the data is discretized/quantized/rounded/truncated.

I need to know if there are general methods, or common methods, to "smooth" the data, remove the discretization, or identify what type of distortion it has (rounding is not the same as truncation or quantization)

I can invent some things, like assuming that data is truncated, and doing interpolations, or using regression to smooth the data, but I don't want to reinvent the wheel and make novice mistakes.

I guess that this is a common problem with measurements, so there must be a set of techniques generally used on experimental physics. Where should I look for them?

  • 6
    $\begingroup$ In many (most, I think) cases the answer is you don't do anything like that. Really. You analyze the data that you have with an understanding of how the measurement apparatus works. $\endgroup$ – dmckee Apr 3 '18 at 16:40
  • $\begingroup$ I agree with @dmckee unless you can give a specific example or reason explaining why you think you need to "smooth" your data. $\endgroup$ – DanielSank Apr 3 '18 at 16:41
  • $\begingroup$ @dmckee; DanielSank I need to calculate derivatives, and use classification algorithms. For example, a part of the curve should be linear, and a wavelet transform should give zero coefficients, but the discretization creates large values for non existent derivatives, or zero derivatives where it should be a slope in the curve. So I need to smooth the data without introducing bias. $\endgroup$ – zexot Apr 3 '18 at 16:45
  • 1
    $\begingroup$ So the real question is what? You want to know specifically how to get derivatives, you want to know how to wavelet transform experimental data, or something else? $\endgroup$ – DanielSank Apr 3 '18 at 16:49
  • 2
    $\begingroup$ "So I need to smooth the data without introducing bias" This is an example of a XY problem. What you need not to massage the data and pretend it is something it isn't, it is to chose an analysis method that deals gracefully with the limits of the data set: do an appropriate global fit and work from the parameters of the fit, or transform to Fourier space and work there. But no one can suggest the right method without knowing what you problem is. $\endgroup$ – dmckee Apr 3 '18 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.