# What are the easiest ways to get data out of a FITS file without a library?

There are FITS libraries for most programming languages (see this list), but FITS is a simple enough format that it should be pretty easy to extract data without the need for such libraries or utilities.

The data portion of an an image is a simple list of pixel values, and of a table is just a catenation of the binary values in each row. The headers are simple enough that it's easy for a person to read the header and see what those rows should be, so my intuition says that it shouldn't be too hard for person to look at the header, and write a short bash script to, say print the rows of a table as an ASCII table that can be processed with awk. After trying it, however, it doesn't seem as easy as I first expected; I generally need to resort to writing (admittedly simple) code.

I'm really looking for something I can run on a system I am unfamiliar with, and will only be working on for a few minutes, such as when I am helping someone else on their computer. So, I'd like to avoid having to install anything, or write anything long or unmemorable enough that would be annoying to write with someone looking over my shoulder.

-

FITS is a simple format but that Flexible there at the front makes it possible for a lot of complexity. I'm not sure there is an "easy" way to get the data out. I think you're going to need at least a little bit of code (Although it would be nice to be wrong).

The problem lies in the fact that there are so many different types of data that could be in the data section. At the first level you have the choice of image data, an ASCII table or a binary table. I think you could get data out of an ASCII table fairly easily once you knew the size of each of the columns from the header since you only have to read off N characters for each column (where N is defined for each column in the header). But that's the only easy one. To parse the binary table you would still have to do the same break up and then do conversions of all the different possible data types (and there are even more data types here than in image data, including vectors of all the possible data types as well).

From your question though, I think you are looking at extracting data from an image array. The header can tell you the data size (8, 16, 32, or 64 bit) and type (integer or floating point) based on the value of the BITPIX keyword. What the header data doesn't tell you is the endianness of the internal data. I don't remember if the FITS Standard prescribes this or not but I don't think it does as I remember having problems moving fits files from big endian (Sparc) machines to little endian (Intel) ones.

So whatever you end up writing to look at the data has to

1. Get the data size and type from the header
2. cycle through the data reading off one data block at a time
3. process the data block properly (reconstruct from the correct endianness and convert from IEEE floating point if necessary) and store or display it.

It's that last step that I'm not sure how easy it is to do simply. I'll admit I've never tried. I always use some preexisting tool (usually fv) to look at the files if I need to do anything more than look at the headers. And when I just have to look at the primary header (or the secondary if the primary one has no data as it is in the files I work with every day), I just read the header by simply doing a 'more' command on the file in an 80 character wide window :).

-
The FITS standard specifies big endian. Some formats that used to be used by IRAF were very close to FITS, but were machine dependent. Some raw files that were produced by the SDSS data acquisition system were little endian, and used a "SIMPLE=FALSE" keyword assignment to indicate that they were not purporting to be real FITS. –  EHN Jun 27 '11 at 23:47
Just to barge-in to the conversation: I'm just wondering if it would be easier to actually install FitsView for any platform you want and finish with it. The process is usually less painful and more interactive for say students than going through the python route. In some systems python is not installed by default but java is there. So you could evan have different copies of FitsView on say USB drive (for each particular OS) and use that by just plugging it in. –  Tigran Khanzadyan Jun 28 '11 at 11:13
@Tigran: I agree. That what I usually do. If fv isn't installed, I just download it and run it. –  dagorym Jun 28 '11 at 13:00
Of course on my own system I use appropriate tools (ds9, fv, TOPCAT, pyfits), but I also want to be able to write a bash script I can hand to someone and reasonably expect to work without any additional effort. Maybe I should just expect that everyone will have a python interpreter, at least. –  EHN Jun 28 '11 at 23:37

hexdump -e '80/1 "%_p" "\n"' myfile.fits


which only half the solution; the other half is getting the actual data out.

If I resort to writing a little python, I can get the output I want. For example, using this command:

hexdump -e '"%_ad\t" 80/1 "%_p" "\n"' astlimits.fits  | less


I can look through the FITS file, and find the byte number of the start of the data I am interested in. (The %_ad starts each line of output with a byte number, the %_p prints each printable character and replaces non-printable ones with a "." Because FITS data must begin at an integer multiple of 2880, the data will start at the start of a line when broken into 80 byte lines.) I can read off the number of rows from the NAXIS2 header keyword of the HDU I am interested in, and the types of each column from the TFROMx keywords.

So, if the data I want starts at byte 5760, the number of rows 5730 (close by coincidence only), and the bytes per row 42 (from NAXIS1), I can use the following python to extract the table I want:

python -c 'import struct
> f = open("astlimits.fits")
> f.seek(5760)
> for i in range(5730): print "\t".join(map(str,struct.unpack(">hhhddhhiiii",f.read(42))))


Less compactly but more readably, this is like running this python script:

import struct
data_start_byte = 5760
num_bytes_in_row = 42   # NAXIS1 from header
num_rows = 5730         # NAXIS2 from header
f = open("astlimits.fits")
f.seek(data_start_byte)
for r in range(num_rows):
row_values = struct.unpack(">hhhddhhiiii", row_bytes)
row_strings = [str(v) for v in row_values]
print "\t".join(row_strings)


If I used a big endian machine, I could do all of this with another hexdump statement. Unfortunately, hexdump won't change endian-ness.

The hardest thing here is remembering how the type codes used in the FITS TFORM values map to the types codes in the first argument of struct.unpack.

But, what I really want is something comparable to hexdump in simplicity.

-
The sample file I used can be found here. –  EHN Jun 27 '11 at 23:37
Other file types (ASCII tables and images) should be slightly easier. FITS heaps are pretty much impossible without lots of code, and fortunately obscure anyway. –  EHN Jun 27 '11 at 23:39

Viewing the headers is easy. (as Eric has shown, although if someone's storing stacks of images, you may need to look for sub-headers). Personally, I just do the same as dagorym, and adjust my window width to 80 columns (or a multiple), and use more or less to read through the headers, unless I'm trying to automate extracting specific headers.

Viewing the data, on the other hand, is going to be near impossible to do reliably without a dedicated FITS reader. The issue isn't even the binary table issue, it's that the FITS standard allows for compression. We've been having a hell of a time with the various FITS readers written for IDL. You basically have to use IRAF's imcopy or CFITSIO's Fpack to write out an uncompressed version, then read it.

SAOimage DS9 reads 'em natively. I'm not an fv user, so I don't know well it supports them. (I would assume it would, but I've never tried)

-
OK, since this is not an answer to the question asked here goes my comment: @Joe Why don't you just install wcstools on your system and make the path accessible in your PATH. This way you can avoid using IRAF for just imcopy. This whole library then can be used easily on any command line or *shell script. –  Tigran Khanzadyan Jun 28 '11 at 20:00
@Tigran : because (1) I didn't know that the WCS tools could do that, and (2) because IRAF is the PI's solution to the problem and (3) IRAF should already be installed as part of SolarSoft (although, parts of the wcstools are too, I think ... I can't remember if parts of wcs fell under ITAR) –  Joe Jun 28 '11 at 22:11
@Joe: WCStools can do many nice things and yes usually some of the data reduction suits do include some version of it. I'm just thinking from the minimalistic point of view. Doing a small operation could be done without IRAF or SolarSoft (which is IDL based). Usually I would have a local ./bin where I would install all such kind of nifty tools and then use them in my CShell scripts, but I suppose it's just a subjective solution. –  Tigran Khanzadyan Jun 29 '11 at 20:10
@Tigran : just a minor point -- SolarSoft is primarily used for distributing IDL, but my boss (who approves the funding for SolarSoft) insists that it's not just for IDL. There's binaries, (Fortran, C), Perl, shell, etc, in it. It's just that the solar community's basically standardized on IDL ... but there's a few groups working to change that ... PDL, SunPy, etc. And lots of people still working in C, Fortran, Matlab, etc. –  Joe Jun 30 '11 at 12:41
@joe: I hope the change would came quick rather than late since IDL license is insanely expensive for us mortals. On the other hand SolarSoft contains many very useful routines that I use almost like 10 years. –  Tigran Khanzadyan Jun 30 '11 at 15:48