Are the results from models considered "data"?

Question

At the AGU, I presented a poster on vocabulary for discussing data systems, and someone left a note at my poster stating:

You have a bias here towards observational data. Need to recognize that a lot of data comes from models and analyses.

And I completely agree; I didn't mention models and the values that come from them at all, and I had defined 'data' in such a way as it only covered observational data:

Values collected as part of a scientific investigation; may be qualified as ‘science data’. This includes uncalibrated values (raw data), derived values (calibrated data), and other transformations of the values (processed data).

... but do scientists consider those resultant values to be 'data'? I've talked to a few scientists over this last week (all in solar or space physics), and everyone of them including those who deal with modeling, were of the option that it was 'model output', but not 'data', (although one made the distinction between the philosphical concept of 'data' being multiple values, but said he didn't consider it to be 'science data' and commented that some of the earth science folks considered the values from their models to be data)

So, the question -- is there a better term to use other than 'model output', and are there fields where 'model output' is considered to be 'data'?

note : there's also the issue of the definitions of raw vs. derived vs. processed data, as there's different definitions used by instrument operators, but just ignore that issue in the definition of 'data' for the time being.

note 2 : I originally asked this on meta, and there were suggetions I move it to the main site; I've edited the text slightly due to a response I had yesterday from my surveying the scientists I work with.

note 3 : to set the context, the original intent was to identify clear or ambiguous terms across science disciplines, but the intended users were the data informatics community (those building data systems for the most part, not necessarily the scientists), but part of the issue is using language that won't offend or confuse the discipline scientists

Answer 1

Cross posted at Chad's blog:

I am a climate scientist. Complex numerical modeling of climate system is a giant neighborhood in the climate science community. There are a great many scientists who work almost entirely within the world of numerical climate models. Many of them refer to the output of climate models as "data". This has been exacerbated by the release of huge archives of climate model output on public websites (the CMIP archive hosted by LLNL) which allows anyone to download and analyze the output from more than 20 climate models. This is accomplished in much the same fashion that large observational groups, such as NASA, share publicly-funded observational data. The CMIP archive has been a great development for opening up the world of climate modeling to any interested party (and to valuable scrutiny by any interested party). But it has, in my opinion, blurred the line between data and simulation output.

My opinion is that the outputs of numerical models should not be referred to as data. Our technical language, particularly in the earth sciences, should distinguish between observations of the one true realization of the earth's climate and the representations of possible (but not necessarily likely or even plausible) climate states that the models spit out.

None of this should be taken as demoting the "status of simulations". I work with models and observations. They both have their place in climate science.

Answer 2

Well, to be really pedantic, almost anything is data. Any measurement or observation is data. I suppose I'd define data as any piece of information, regardless of how it was obtained or whether it is valid.

Of course, I think your question was really getting at what would be considered evidence, in the context of providing support or opposition to a theoretical model. I'd say that the answer is very situation dependent. Using simulation results as an example since it was in your question, I would certainty take simulation results in support of some theory, but only on the condition that the simulation has been very well validated against experiment. The details of how and why the simulation results are favorable to the theory is also certainly important.

Ultimately, all science must be validated by experiment or observation, but that can sometimes be a long journey, and there is definitely a place for other sorts of data along the way.

Answer 3

I'm an experimenter, so I stand pretty strongly with the idea that "data" comes from experiments. But life is rarely that easy and there are some complicating factors.

In particle physics the main complicating factor comes in the form of lattice QCD.

It works like this: you evaluate the time-dependent, field-theoretical potentials on a lattice (and apply some clever magic due to Peter LePage and others to take care of the high frequency stuff that would otherwise make the calculation too big to handle).

Exactly what potential you evaluate is a matter of some choice as there are an number of free parameters in the theory. Of course, you also get to choose the situation to be simulated. Now, because this is a QCD calculation there are very few situations where you already know the answer. That's what make the technique useful. But it also means you can not anticipate the outcome of an calculation until you run the code.

Lattice QCD people sometimes speak of this as being "an experiment" and of the results as "data". At least they do among themselves and when talking to more traditional theorists.

I accept this as a bit of a gray area in the "data" versus "theory" space.

On one hand they don't know the results until they crunch the numbers and then they have to live with them. The results are generally repeatable, and sometimes takes some work to compare to "pure" theory.

On the other it is calculation and they can (and do) change the micro-physics assumptions that go into the models.

---

Another gray area in my mind concerns ("real" experimental data) that have been transformed in a model to make them comparable to another data set. (This is separate from analysis, for instance in nuclear or particle physics cross-sections, form-factors and the like will sometimes be "adjusted" by factors known from models to allow them to be compared to results from other experiments done at different kinematics.) Data? Well, mostly, but dependent on the theory, too.

Also, how much selecting and correcting are you willing to allow before a particular set of data has to be taken with a "if they did all the subsequent calculations right" grain of salt.

Answer 4

This is a politically loaded question, which is probably why you got the response you did at the AGU, where there are people who work on climate change issues. There is a concerted effort in some quarters to cast doubt on the science showing that the Earth is warming, and predicting the effects of climate change. As these predictions rely heavily on simulations, one part of this effort is an attempt to disparage models and simulations as having lesser status, scientifically.

As for the actual status of models and simulations, that varies from (sub)field to (sub)field, more or less in accordance with how difficult it is to interpret experimental or observational data. My own field of experimental Atomic, Molecular, and Optical physics has a fairly clear divide between experiment and theory, largely because the experiments we do are relatively unambiguous: an atom either absorbs light or doesn't, or it's either in this position or that one. We do need simulations to compare to some experiments, but there's never much question that those are theory, and not part of the experiment.

When you get to nuclear and particle physics, where the detectors are the size of office buildings, the line gets a little fuzzier. The systems they use to detect and identify the products of collisions between particles are so complicated that it's impossible to interpret what happens without a significant amount of simulation. As a result, experimental nuclear and particle physicists spend a great deal of time generating and analyzing simulated data, in order to account for issues of detector efficiency and so on. I don't think they would call these results "data" per se, but computational simulation is an absolutely essential part of experimental physics in those fields, and those simulations are accorded more status than they would be in AMO physics.

Things get even more complicated when you get to parts of physics that are fundamentally observational rather than experimental. If you're a particle physicist, you can repeat your experiments millions or billions of times, and build up a very good statistical understanding of what happens. If you're an astrophysicist or a geophysicist, you only get one data run-- we have only one observable universe, and only one Earth within it to study. You can't rewind the history of the observable universe and try it again with slightly different input parameters. Unless you do it in a simulation.

My outsider's understanding of those fields is that simulation and modeling is accorded a much higher status than in my corner of physics, just out of necessity. If you want to use a physical model to explain some geological or astrophysical phenomenon, the only way you can really do it is by running a whole lot of simulations, and showing that the single reality we observe is a plausible result of your models. Correctly interpreting and establishing correspondences between simulations and observation is a subtle and complicated business, and constitutes a huge proportion of the work in those communities.

I don't know that many astrophysicists, and even fewer geophysicists, so I don't know the terminology they use. My impression of the astrophysics talks I've seen is that they wouldn't put such simulation results on the same level as observational data or experiments, but then my sample isn't remotely representative. It may well be that there are fields in which model results are deemed "data" in the local jargon.

Answer 5

A great quote I heard at the American Chemical Society conference last year went something like, "All experimentalists trust each other's results, but not their own; All simulators distrust each other's results, but not their own." The reason that happens is that experimentalists rigorously describe their methods, use stock reagents, and even publish the model numbers of their instruments. It's easy, at least in principle, to replicate their work, so many people trust that they don't have to. Simulators don't do that, partly because there aren't standardized ways to do it(yet), partly because the community hasn't demanded it(yet), and partly because they don't want to give away their software. It is difficult, even in principle, to replicate the results produced by a simulator, thus many feel that they can't trust the results. That's changing -- perhaps when it does more people will regard the results of simulations as "data".

For my part, I'm in computational biophysics, doing simulations of protein folding, and I do regard results produced from my simulations ("in silico") as data, and the simulations themselves as experiments. I try to maintain the view that the data and experiments are about the model, however, especially when talking to experimentalists, and even more so when talking to people with a medical or biological background (they tend to be somewhat hostile to simulation, in my experience). If I make claims about the relationship between the model, the data from the model, and the results one would obtain by doing an analogous experiment in vitro I have to be very careful about validation of the model in as many independent ways as possible, and in careful vetting of the methods.

For example, my measurement of secondary structure content of an ensemble of protein structures is not fundamentally different from that of an experimentalist, except that I have a lot more information available and have to throw some of it away to produce an average measurement that is commensurable to the one obtained by the experimentalist using circular dichroism spectroscopy (and some interpretation based on a theoretical model). I'm still not going to claim that my measurement is the correct result for the protein of interest unless I'm very certain that the model and the free parameters used in the relevant simulation were the correct ones to use for the situation, but I would give the experimenter the benefit of the doubt that their results are probably the correct ones.

And finally, if I was to be very careful about the word "data", I would say that the only data produced by my simulations are the coordinates of the atoms themselves. Everything else is a derived measurement using those coordinates. Why? Because I could recreate all the measurements from the coordinates, but I couldn't recreate the coordinates from the measurements.

Answer 6

Data from modelling is still data; the only thing is that it describes the model, not the real phenomenon.

Answer 7

I know that this is an old question, but I just wanted to add that in Earth systems science I've often heard people use the term "reanalysis data" to refer to model output when it's used as input data to an analysis technique. (I think this might be more specific than what you're asking for though, as I believe it refers to the output of a specific type of model, namely grid-based global models that have been fit to coarser input data. But still it seemed worth mentioning.)