Physics Stack Exchange is a question and answer site for active researchers, academics and students of physics. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've been following the news around the work they are doing at the LHC particle accelerator in CERN. I am wondering what the raw data that is used to visualize the collisions looks like. Maybe someone can provide a sample csv or txt?

share|cite|improve this question
Keep in mind that data from the LHC come in terrabytes and the processing needs have created a whole new way of data handing, called GRID. . It describes the handling of the data. – anna v Jun 23 '11 at 6:06
CERN recently released 300 TB of data: – opensourcechris Apr 23 at 23:55

Different pieces of equipment will produce somewhat different looking data, but typically it consists of voltages defined as a function of time. In some cases (spark chambers, for example) the "voltage" is digital, and in others it is analog.

Traditionally, the time series for the data is slower than the times required for the (almost light speed) particles to traverse the detector. Thus one had an effective photograph for a single experiment. More modern equipment is faster but they still display the data that way. Here's an LHC example:

enter image description here

In the above, the data has been organized for display according to the shape and geometry of the detector. The raw data itself would be digitized and just a collection of zeroes and ones.

There are typically two types of measurements, "position" and "energy". The position measurements are typically binary, that is, they indicate that a particle either came through that (very small) element or did not. In the above, the yellow lines are position measurements.

Note that some of the yellow lines are curved. Actually all of them are curved at least some. This is because there is a strong magnetic field. The curvature of the particle tracks helps determine what particles they are. For example, given the same speed and charge, a heavier particle will run straighter.

The radius of curvature is given by:
$$r = \frac{m\gamma E}{pB}$$ where $\gamma = 1/\sqrt{1-(v/c)^2}$ is the Lorentz factor, $E$ is the energy, and $p$ is the momentum. This helps determine the particle type and energy.

Energy measurements are generally analog. In them, one gets an indication of how much energy was deposited by the particle as it went by. In the above, the light blue and red data are energy measurements. For these measurements, one doesn't get such a precise position, but the amplitude is very accurate.

share|cite|improve this answer
Note that this view isn't even remotely "raw". Considerable reconstruction and tracking has already been done. – dmckee Jun 22 '11 at 20:49
This is just amazing science, thank you. I would still like to see some of the data generated by the sensor elements and also possibly at various reconstruction/aggregation stages. Can you help with that? – opensourcechris Jun 22 '11 at 20:56
Another comment: "Typically the time series for the data is slower than the times required for the (almost light speed) particles to traverse the detector." is mostly incorrect. Most detector elements have time resolution at the 1--10 ns scale which corresponds to order of 1--10 feet (.3--3 meter) at $c$. In the case of a big collider detector package as pictured here the instrument is 10 meters or more on a side. In many cases time of flight is used to help with particle ID. – dmckee Jun 22 '11 at 21:10
@opensourcechris this would be an exercise in futility. The raw data are useless without the metadata, including the contents of logs by the shift babysitting the detectors. The for profit niches happen when the detectors are built. A lot are outsourced to industry . There is no profit from data gathering to be shared out. The institutes even pay for the publications. – anna v Jun 23 '11 at 5:34
@opensourcechris I think generally speaking the main thing preventing institutions from releasing data is the shear amount of bandwidth it would take to provide it to everyone. The LHC, for example, produces one petabyte of raw data every second. Automatic filters take out the noise and not-useful data and only a small fraction is recorded. At the end of these cuts, only 25 petabytes is recorded annually. This is a huge amount of data, only 20% or so of it is stored at CERN and the rest is distributed to affiliated organizations. – Benjamin Horowitz Jun 24 '11 at 6:01

Years ago, as a grad student in particle physics, I used to work on the PHENIX experiment at BNL. Before I had shown up (I think near the end of run 2) the main data structure used for analysis was called a "tuple". Tuples were pretty much like the lists used today in Python with a bit more structure to make access faster and contained the actual data corresponding to what we called an "event" (something interesting that happened in the detector which was captured by the various subsystems and written eventually into a tuple). Unfortunately tuples were generally just too large and one needed to analyze a smaller subset of the entries in the tuples -- so micro-tuples were born and then shortly afterwards nano-tuples.

There were different types of nano-tuples defined and used by the various working groups on the experiment which had different subsets of the original tuples. Which type of nano-tuple you used depended on the analysis you were trying to do and roughly corresponded to the working group you were in. In my case this was heavy flavor where I was studying charm.

So a nano-tuple might look like this:

(x_1, x_2, ..., x_n)

where the x_i would be all the different quantities of interest associated with the event: transverse momentum, energy deposited in the EM-cal, blah, blah, blah..

In the end the data analysis revolved around the manipulation of these nano-tuples and amounted to:

  1. Put in a request with the data guys to get raw data collected by the different subsystems in the form of nano-tuples.
  2. Wait a couple days for the data to show up on disk since it was a huge set of data.
  3. Loop over the events (nano-tuples) filtering out the stuff you weren't interested in (usually events associated with pions)
  4. Bin the data in each entry of the tuple
  5. Overlay the theoretical prediction of these distributions on top of what you extracted from the tuple
  6. Make your statement about what was going on. (confirmation of theory, conjecture about disagreement, etc..)

The truth is that we rarely looked at the RAW, raw data streaming out of the detector unless you were on shift and part of the data acquisition system had stopped operating for some reason. But in that case the data was pretty meaningless when you looked at it. You'd be more concerned that the data wasn't flowing. However if you were one of the people responsible for maintaining a subsystem (say, EM-cal) then you'd probably be doing calibration on a regular basis and regularly looking over raw data from your particular subsystem to tune the calibration and make the raw data analyzable.

Mostly the raw data was only meaningful for the subsystem you had a responsibility to and looking at all the raw data from all the subsystems as a whole wasn't really done. I don't think anyone had that kind of breadth across all the different subsystems...

Regarding the data for the visualizations you asked about: I believe these were specially defined nano-tuples which had entries from enough of the subsystems to allow for reconstruction and the final visualization (pretty pictures) but I'm 99% sure the visualizations weren't created from the "raw" data. Rather they were done using these nano-tuples.

If you poke around the PHENIX website you can see some pretty fancy animations (at least fancy for back then) of collisions in the detector. Mostly these pics and movies were part of a larger experiment wide PR effort. They were made by a guy named Jeffery Mitchel and you should email him to find out more details on the format of the data he used ( Everyone was talking about the LHC back then (2004 or 2005ish) and most of them have long since moved on so you can probably get more insight into the "raw" data created by the LHC today and used for those visualizations if you ask someone like him directly.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.