What are the easiest ways to get data out of a FITS file without a library?

Question

There are FITS libraries for most programming languages (see this list), but FITS is a simple enough format that it should be pretty easy to extract data without the need for such libraries or utilities.

The data portion of an an image is a simple list of pixel values, and of a table is just a catenation of the binary values in each row. The headers are simple enough that it's easy for a person to read the header and see what those rows should be, so my intuition says that it shouldn't be too hard for person to look at the header, and write a short bash script to, say print the rows of a table as an ASCII table that can be processed with awk. After trying it, however, it doesn't seem as easy as I first expected; I generally need to resort to writing (admittedly simple) code.

I'm really looking for something I can run on a system I am unfamiliar with, and will only be working on for a few minutes, such as when I am helping someone else on their computer. So, I'd like to avoid having to install anything, or write anything long or unmemorable enough that would be annoying to write with someone looking over my shoulder.

Answer 1

FITS is a simple format but that Flexible there at the front makes it possible for a lot of complexity. I'm not sure there is an "easy" way to get the data out. I think you're going to need at least a little bit of code (Although it would be nice to be wrong).

The problem lies in the fact that there are so many different types of data that could be in the data section. At the first level you have the choice of image data, an ASCII table or a binary table. I think you could get data out of an ASCII table fairly easily once you knew the size of each of the columns from the header since you only have to read off N characters for each column (where N is defined for each column in the header). But that's the only easy one. To parse the binary table you would still have to do the same break up and then do conversions of all the different possible data types (and there are even more data types here than in image data, including vectors of all the possible data types as well).

From your question though, I think you are looking at extracting data from an image array. The header can tell you the data size (8, 16, 32, or 64 bit) and type (integer or floating point) based on the value of the BITPIX keyword. What the header data doesn't tell you is the endianness of the internal data. I don't remember if the FITS Standard prescribes this or not but I don't think it does as I remember having problems moving fits files from big endian (Sparc) machines to little endian (Intel) ones.

So whatever you end up writing to look at the data has to

1. Get the data size and type from the header
2. cycle through the data reading off one data block at a time
3. process the data block properly (reconstruct from the correct endianness and convert from IEEE floating point if necessary) and store or display it.

It's that last step that I'm not sure how easy it is to do simply. I'll admit I've never tried. I always use some preexisting tool (usually fv) to look at the files if I need to do anything more than look at the headers. And when I just have to look at the primary header (or the secondary if the primary one has no data as it is in the files I work with every day), I just read the header by simply doing a 'more' command on the file in an 80 character wide window :).

Answer 2

Viewing the headers is easy. (as Eric has shown, although if someone's storing stacks of images, you may need to look for sub-headers). Personally, I just do the same as dagorym, and adjust my window width to 80 columns (or a multiple), and use more or less to read through the headers, unless I'm trying to automate extracting specific headers.

Viewing the data, on the other hand, is going to be near impossible to do reliably without a dedicated FITS reader. The issue isn't even the binary table issue, it's that the FITS standard allows for compression. We've been having a hell of a time with the various FITS readers written for IDL. You basically have to use IRAF's imcopy or CFITSIO's Fpack to write out an uncompressed version, then read it.

SAOimage DS9 reads 'em natively. I'm not an fv user, so I don't know well it supports them. (I would assume it would, but I've never tried)

Answer 3

On most UNIX/Linux systems, you can read the header like this:

hexdump -e '80/1 "%_p" "\n"' myfile.fits

which only half the solution; the other half is getting the actual data out.

If I resort to writing a little python, I can get the output I want. For example, using this command:

hexdump -e '"%_ad\t" 80/1 "%_p" "\n"' astlimits.fits  | less

I can look through the FITS file, and find the byte number of the start of the data I am interested in. (The %_ad starts each line of output with a byte number, the %_p prints each printable character and replaces non-printable ones with a "." Because FITS data must begin at an integer multiple of 2880, the data will start at the start of a line when broken into 80 byte lines.) I can read off the number of rows from the NAXIS2 header keyword of the HDU I am interested in, and the types of each column from the TFROMx keywords.

So, if the data I want starts at byte 5760, the number of rows 5730 (close by coincidence only), and the bytes per row 42 (from NAXIS1), I can use the following python to extract the table I want:

python -c 'import struct
> f = open("astlimits.fits")
> f.seek(5760)
> for i in range(5730): print "\t".join(map(str,struct.unpack(">hhhddhhiiii",f.read(42))))

Less compactly but more readably, this is like running this python script:

import struct
data_start_byte = 5760
num_bytes_in_row = 42   # NAXIS1 from header
num_rows = 5730         # NAXIS2 from header
f = open("astlimits.fits")
f.seek(data_start_byte)
for r in range(num_rows):
    row_bytes = f.read(num_bytes_in_row)
    row_values = struct.unpack(">hhhddhhiiii", row_bytes)
    row_strings = [str(v) for v in row_values]
    print "\t".join(row_strings)

If I used a big endian machine, I could do all of this with another hexdump statement. Unfortunately, hexdump won't change endian-ness.

The hardest thing here is remembering how the type codes used in the FITS TFORM values map to the types codes in the first argument of struct.unpack.

But, what I really want is something comparable to hexdump in simplicity.