How can I estimate a confidence interval for experimental results with only one run?

Question

I'm planning an experiment, but due to cost and other circumstances, there will only be one realization of it. So the plan is to collect as many different types of data as we can with a variety of techniques and sift through it. However, I'm not sure how to address quantifying or estimating the uncertainty in any given set of final results.

I can run any number of tests on subsets or components of the experiment and the data collection using representative conditions -- maybe scaled down, or maybe isolating certain aspects, etc..

If we assume everything is deterministic, how can I take uncertainty from these proxy/surrogate experiments using components to estimate the uncertainty (or to establish a confidence interval) on the final data from the full experiment?

As an example, let's say that my experiment is to run a large reacting flow experiment. I will have many different cameras aimed at it measuring different things (via various filters, lasers, and so on) and I will have many different pressure sensors and temperature sensors. I cannot run the full experiment more than once, but I can test any of the measurement systems on smaller, more representative problems:

1. Cameras can be set up to look at simple flames, like a Bunsen burner;
2. Pressure sensors can be hooked up to tubes with speakers;
3. Temperature sensors can be placed in various heated flows.

For any of those simple setups, I can vary anything I want and run as many times as I want. So, I could run my camera with a large range of possible settings and possible optical paths and get the uncertainty in a given measurement due to those changes in settings. I could run the pressure sensors over a range of frequencies, amplitudes, sampling rates, etc. and measure the uncertainty there.

But, how can those uncertainties measured from simpler experiments inform the confidence in the full experiment? Because the full system is very complicated, there is no complete set of model equations to propagate uncertainty through. Since I cannot run more than once, I cannot vary settings and evaluate sensitivities. I can only run proxy experiments and have to somehow use that information to estimate a confidence or uncertainty in the full experiment.

Answer 1

Its seems to me as a rather complex experiment that requires serious probabilistic modeling rather than a simple estimate of confidence intervals.

Simplistic view
Confidence intervals are a tool of frequentist statistics which rely on making multiple measurements. For example, given a set of repeated measurements $\{x_1, x_2, ..., x_n\}$ and assuming Gaussian statistics, the mean and the variance can be estimated as $$ \hat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i,\\ \hat{\sigma^2} = \frac{1}{n-1}\sum_{i=1}^n(x_i - \hat{\mu})^2, $$ where the hats indicate that these are estimators rather then the actual values. With only one measurement the estimate of mean is just the measured value, whereas the unbiased estimate variance is non-existent, since $n-1=0$. Even if one agrees to use a biased estimate of variance (with $n$ instead of $n-1$), with only one measurement it is identically equal to zero. Without the variance one cannot find the confidence intervals, which is probably the basis for this question.

Maximum likelihood approach
However, when multiple quantities $\{q_1, q_2, ..., q_n\}$ are measured, which are all dependent on a smaller set of parameters $\theta$, one can construct the likelihood of observing this set of quantities $$ P(q_1, q_2, ..., q_n|\theta) $$ Maximizing this likelihood gives the estimator for the parameters, whereas calculating Fisher information gives the estimate of the variance, which, in principle, allows calculating confidence intervals (although statisticians might disagree that calculating confidence intervals is appropriate in this case).

Bayesian approach
A problem with a single measurement is probably more naturally treated in the Bayesian framework, using the Bayes theorem to define the probability distribution of the confidence intervals, given the observed set of diverse measurements $$ P(\theta|q_1, q_2, ..., q_n) \propto P(q_1, q_2, ..., q_n|\theta)P(\theta), $$ where the prior probability, $P(\theta)$ reflects the initial knowledge about the parameter variation, e.g., due to the known error/precision of the equipment. One can then trivially estimate credibility intervals, which are the equivalent of the frequentist's confidence intervals for the Bayesian case (and in fact are easier to interpret). Let me note however that using the Bayesian versus the likelihood framework is a matter of convenience here.

Finally, you might get more answers, if you post a question in the statistics forum (Cross Validated), although your description of the experimental details might have to be streamlined/simplified.

Answer 2

Estimation of confidence intervals for quantities from a single complex situation requires that you have a reliable statistical model of the whole situation relating all the data that you have form the various experiments done. Under suitable assumptions, the unknown parameters and error covariance matrices entering the model can be estimated from the avaliable data, which gives an assessment of the accuracy of a single experimental result form the totality of measured data.

The proptoype of such an estimation method is the restricted maximum likelihood (REML) procedure, about which there is a large literature and overwhelming positive evidence for its usefulness. For example, REML is the basis of the commercial methods for animal breeding, where the quantities of interest are breeding values of key animals to be used for the planning of breeding and the model relations come from the Mendelian laws. See, e.g., my paper

• A. Neumaier and E. Groeneveld, Restricted maximum likelihood estimation of covariances in sparse linear models, Genet. Sel. Evol. 30 (1998), 3-26.

REML is most developed for models linear in the parameters (but nonlinear in the covariance components). Extensions to nonlinear models are also possible but are much less explored.