How to deal with outliers in experimental physics? In experimental physics, is it generally considered a good practice to eliminate outliers? I was taught that, when doing a linear regression, I should prune all the data that doesn't fit within 3 sigma (standard deviation) and recalculate the new SD, etc. which I think is a very bad practice.
Sometimes a variable linearly depends on another variable but such a linearity is reached in practice after a certain amount of time. The measurements however are taken from say $t=0$ up to T. The linearity holds roughly for $0 < t_1 < t_2 < T$ (for instance if a machine gets hotter and hotter until the linearity breaks, and was too cold at the beginning for the linearity to hold). What is a good practice to prune the outliers data and consider only the linear region? I was taught to manually get rid of all the data that doesn't look linear to my eyes, but I think that's a very bad practice. I had thought to maybe calculate the coefficient of determination ($R^2$) and keep the points that makes it stay above a particular subjective threshold such as 0.99, but in the end I think that's also a bad practice. 
So I'm wondering whether there are standard ways to deal with outliers in the 2 cases mentioned above.
 A: The only way to avoid using ad-hoc methods and do things "right" from the start is to get to a good model that gives you the probability of observing a certain data set (including the outliers) given what the true state of the system that you want to measure, is. You can then compute the probability distribution for your observable you want to get to, given whatever data has been measured (using Bayes' theorem). But in practice it's usually infeasible to do this fully on a first principles basis, so you'll still end up with ad-hoc assumptions that go into the model that describes the measured data given the real data.
You can also go about this using simulations. Here you consider a set of theoretical models, none of which has to be the "correct one". You then generate simulated experimental data (including outliers) from each theoretical model (and for each model you should generate multiple data sets). You then analyze the generated data to get to the estimated model parameters. This then yields a lot of data for validating your approach, you can see if the analysis method is biased in some way. You can then tweak the analysis method to correct for the bias and then repeat the whole process to see if the modification has reduced the problem. You can't do this on the basis of only the original data set, because adjustments based on stochastic outcomes can lead to artifacts due to statistical fluctuations.
