0
$\begingroup$

I am fitting a linear polynomial to some data and I have derived the errors for each of the best-fit parameters from the covariance matrix. I would expect these errors to correspond to a $1\sigma$ 68% confidence interval, but I am finding this is not the case.

If I instead do a grid search, where I freeze one of the two parameters and grid the other, searching for the parameter values corresponding to $\chi^{2}_{\mathrm{min}} + 1$, I get a smaller error interval than from the least squares errors. According to Bevington (Data Reduction and Error Analysis for the Physical Sciences), a single-parameter 68% confidence interval is given by parameter values that increase the $\chi^{2}$-value from $\chi^{2}_{\mathrm{min}}$ to $\chi^{2}_{\mathrm{min}}+1$.

I have tested this with multiple codes and they all give the same results. Can someone help me to understand this behavior?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Better on Cross Validated? $\endgroup$
    – dmckee --- ex-moderator kitten
    Commented May 24, 2016 at 20:24
  • $\begingroup$ Is your covariance matrix diagonal? If it's not, then your two parameters are correlated. Your second analysis (grid search) seems to be implicitly assuming that your two parameters are uncorrelated. $\endgroup$
    – Paul T.
    Commented May 25, 2016 at 12:30
  • 1
    $\begingroup$ @dmckee yes, I think this is off topic here $\endgroup$
    – David Z
    Commented May 25, 2016 at 12:36

2 Answers 2

Reset to default
2
$\begingroup$

I think what you have described is not the correct method to estimate a 68% confidence interval in one parameter of interest.

The error is in freezing the other parameter when minimising chi-squared.

A better procedure to evaluate the uncertainty in parameter 1, is to evaluate the minimum chi-squared for a set of values of parameter 1, whilst allowing parameter 2 to vary. The best fit parameter 1 value is at the global minimum, whilst an estimate of its uncertainty is provided by where the chi-squared increases by 1 from this value, but not necessarily at the same value of parameter 2.

If you freeze parameter 2 at its best fit value, you will underestimate the uncertainty in parameter 1. The reasons for this is that the locus of least chi-squared fit in a parameter 1 vs parameter 2 space is (in general) inclined with respect to these axes.

The best way to do it is to evaluate chi-squared over all the parameter space and then find the projection of the chi-squared+1 contour onto each of the parameter axes. A picture may say a thousand words. The red arows show how you tried to do it. The blue limits show my (economical) first way of trying to improve the estimate and then the black arrows show the projection of the error ellipse onto the x-axis.

Error ellipse

Improve this answer
$\endgroup$
3
  • $\begingroup$ In other words, for 68% in 1D find the (possibly discontiguous ) region in $x$ where$\min_{y,z\cdots}\chi^2(x, y, z,\cdots) - \min_{x,y,z\cdots}\chi^2(x, y, z,\cdots) ≤ 1$ $\endgroup$
    – innisfree
    Commented May 25, 2016 at 13:38
  • $\begingroup$ Thanks! This makes sense to me. It seems the error from least squares correctly gives you the 1$\sigma$ projection of the error ellipse on the parameter axis for a line when the ellipse is inclined as a result of correlations between parameter 1 and parameter 2. It looks like you can also get the errors from projecting the error ellipse onto the parameter axes using the covariance matrix. How would you derive 1 sigma confidence intervals if you have more than 2 parameters (say, 5 parameters) and several of them are correlated with each other? $\endgroup$
    – grover
    Commented May 27, 2016 at 4:20
  • $\begingroup$ @grover The expensive way is to calculate an entire grid and project along each axis. Monte Carlo might be cheaper. Or you increment one param, minimise chisq wrt the rest and use this 1d plot to find where chisq increases by +1. This is computationally cheap and roughly correct. Note that it is only chisq+1 if you are interested in only one parameter. $\endgroup$
    – ProfRob
    Commented May 27, 2016 at 6:06
2
$\begingroup$

In the case of a 2-parameter fit, the 68% confidence space is typically an ellipse, not necessarily aligned with the axes. If you want to figure out the size of the ellipse, you should find the orientation of the ellipse, not just the intercepts of the ellipse with a horizontal and vertical line through its center.

Example: fit $y=a+b x$ for a large set of data points clustered around $(x,y)=(1,1)$. The quality of the fit will be pretty similar for $(a,b)=(1,0)$ or $(0,1)$. But if you change $a$ without changing $b$ in the opposite direction (this is what you did), you will very quickly get a bad fit. But this doesn't tell you much about the uncertainty in $a$.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The procedure I use is to iteratively change one parameter without changing the other. Then, I search for $\chi^{2}_{\mathrm{min}}$ and update the value. This process is repeated for each parameter until $\chi^{2}_{\mathrm{min}}$ does not change and you obtain a global minimum. This guarantees you are getting the best fit by doing an exhaustive search. Then, you can derive confidence intervals from looking at parameter values corresponding to $\chi^{2}_{\mathrm{min}} + 1$. My question is why is this different from the errors one derives from the least squares covariance matrix? $\endgroup$
    – grover
    Commented May 24, 2016 at 20:21
  • $\begingroup$ That what the answer tells you: because the error ellipse is not guaranteed to be axis-aligned. $\endgroup$
    – dmckee --- ex-moderator kitten
    Commented May 24, 2016 at 20:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.