6
$\begingroup$

I'm planning an experiment, but due to cost and other circumstances, there will only be one realization of it. So the plan is to collect as many different types of data as we can with a variety of techniques and sift through it. However, I'm not sure how to address quantifying or estimating the uncertainty in any given set of final results.

I can run any number of tests on subsets or components of the experiment and the data collection using representative conditions -- maybe scaled down, or maybe isolating certain aspects, etc..

If we assume everything is deterministic, how can I take uncertainty from these proxy/surrogate experiments using components to estimate the uncertainty (or to establish a confidence interval) on the final data from the full experiment?

As an example, let's say that my experiment is to run a large reacting flow experiment. I will have many different cameras aimed at it measuring different things (via various filters, lasers, and so on) and I will have many different pressure sensors and temperature sensors. I cannot run the full experiment more than once, but I can test any of the measurement systems on smaller, more representative problems:

  1. Cameras can be set up to look at simple flames, like a Bunsen burner;
  2. Pressure sensors can be hooked up to tubes with speakers;
  3. Temperature sensors can be placed in various heated flows.

For any of those simple setups, I can vary anything I want and run as many times as I want. So, I could run my camera with a large range of possible settings and possible optical paths and get the uncertainty in a given measurement due to those changes in settings. I could run the pressure sensors over a range of frequencies, amplitudes, sampling rates, etc. and measure the uncertainty there.

But, how can those uncertainties measured from simpler experiments inform the confidence in the full experiment? Because the full system is very complicated, there is no complete set of model equations to propagate uncertainty through. Since I cannot run more than once, I cannot vary settings and evaluate sensitivities. I can only run proxy experiments and have to somehow use that information to estimate a confidence or uncertainty in the full experiment.

$\endgroup$
8
  • $\begingroup$ Your question is not clear to me: are you asking how to assign an uncertainty to a measurement that you can only take once or are you asking how to propagate the uncertainty of a measurement to another one? Maybe try to specify a little better what do you mean by "proxy experiments". $\endgroup$
    – Noumeno
    Oct 22, 2020 at 21:59
  • $\begingroup$ @Noumeno I added an example with what I meant by "proxy experiments" -- but let's not get too hung up on those specific examples. I could have also picked something like the LIGO experiments, where they couldn't possibly run multiple black hole collisions to determine uncertainties and would have needed something smaller, controllable, but representative to establish confidence. $\endgroup$
    – tpg2114
    Oct 22, 2020 at 22:11
  • $\begingroup$ So I guess to answer your question in the comment -- I want to assign uncertainty or confidence to a measurement I will only take once. But I can run a whole bunch of other, simpler measurements as many times as I want to help me assign that uncertainty to the real measurement. $\endgroup$
    – tpg2114
    Oct 22, 2020 at 22:13
  • $\begingroup$ First, this question is very broad and I doubt there will be an answer in the form of a recipe you can follow, probably the best you can hope for is literature in a related field. From what you have said, you have a very complicated setup that you can only run once (so you can't build an empirical understanding) and you lack a good model for it, therefore I suspect the systematic uncertainties due to a lack of understanding of the experiment will matter much more than sensor noise. What are the key variables and what are you trying to measure? $\endgroup$
    – Andrew
    Nov 2, 2020 at 3:52
  • $\begingroup$ @Andrew I agree it is broad, but ideally there is a mathematical framework to handle it -- I used one example, but there's many others that could be made. We only get one chance with an impactor probe, for example. I can build models for aspects of the system, but I can't build a model that includes all the non-linear interactions of things that could occur (and the experiment is actually the model for the real system). I could also just as easily ask the question about a computer simulation, where it would take months and all of our resources to run once but I need to know how reliable it is $\endgroup$
    – tpg2114
    Nov 2, 2020 at 4:03

2 Answers 2

2
+350
$\begingroup$

Its seems to me as a rather complex experiment that requires serious probabilistic modeling rather than a simple estimate of confidence intervals.

Simplistic view
Confidence intervals are a tool of frequentist statistics which rely on making multiple measurements. For example, given a set of repeated measurements $\{x_1, x_2, ..., x_n\}$ and assuming Gaussian statistics, the mean and the variance can be estimated as $$ \hat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i,\\ \hat{\sigma^2} = \frac{1}{n-1}\sum_{i=1}^n(x_i - \hat{\mu})^2, $$ where the hats indicate that these are estimators rather then the actual values. With only one measurement the estimate of mean is just the measured value, whereas the unbiased estimate variance is non-existent, since $n-1=0$. Even if one agrees to use a biased estimate of variance (with $n$ instead of $n-1$), with only one measurement it is identically equal to zero. Without the variance one cannot find the confidence intervals, which is probably the basis for this question.

Maximum likelihood approach
However, when multiple quantities $\{q_1, q_2, ..., q_n\}$ are measured, which are all dependent on a smaller set of parameters $\theta$, one can construct the likelihood of observing this set of quantities $$ P(q_1, q_2, ..., q_n|\theta) $$ Maximizing this likelihood gives the estimator for the parameters, whereas calculating Fisher information gives the estimate of the variance, which, in principle, allows calculating confidence intervals (although statisticians might disagree that calculating confidence intervals is appropriate in this case).

Bayesian approach
A problem with a single measurement is probably more naturally treated in the Bayesian framework, using the Bayes theorem to define the probability distribution of the confidence intervals, given the observed set of diverse measurements $$ P(\theta|q_1, q_2, ..., q_n) \propto P(q_1, q_2, ..., q_n|\theta)P(\theta), $$ where the prior probability, $P(\theta)$ reflects the initial knowledge about the parameter variation, e.g., due to the known error/precision of the equipment. One can then trivially estimate credibility intervals, which are the equivalent of the frequentist's confidence intervals for the Bayesian case (and in fact are easier to interpret). Let me note however that using the Bayesian versus the likelihood framework is a matter of convenience here.

Finally, you might get more answers, if you post a question in the statistics forum (Cross Validated), although your description of the experimental details might have to be streamlined/simplified.

$\endgroup$
2
  • $\begingroup$ I am totally on board with Bayesian methods, but just a note that the standard approach is to use MCMC or nested sampling which requires evaluating the likelihood, which in turn means having a parameterized forward model of the experiment (since you need to calculate the probability of the data given the parameters). This is something the OP said they did not have. $\endgroup$
    – Andrew
    Nov 2, 2020 at 10:42
  • $\begingroup$ @Andrew I don't think they need a full model of an experiment, but rather a model of measurement: e.g., if the same object is photographed from different angles by different cameras. In any case, I doubt that anything of comparable complexity can be deal with non-parametrically. $\endgroup$ Nov 2, 2020 at 10:49
2
$\begingroup$

Estimation of confidence intervals for quantities from a single complex situation requires that you have a reliable statistical model of the whole situation relating all the data that you have form the various experiments done. Under suitable assumptions, the unknown parameters and error covariance matrices entering the model can be estimated from the avaliable data, which gives an assessment of the accuracy of a single experimental result form the totality of measured data.

The proptoype of such an estimation method is the restricted maximum likelihood (REML) procedure, about which there is a large literature and overwhelming positive evidence for its usefulness. For example, REML is the basis of the commercial methods for animal breeding, where the quantities of interest are breeding values of key animals to be used for the planning of breeding and the model relations come from the Mendelian laws. See, e.g., my paper

REML is most developed for models linear in the parameters (but nonlinear in the covariance components). Extensions to nonlinear models are also possible but are much less explored.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.