NOTICE: This Legacy journal article was published in Volume 1, May 1992, and has not been updated since publication. Please use the search facility above to find regularly-updated information about this topic elsewhere on the HEASARC site.

FITSIO and other FITS-related activities

at the HEASARC

William Pence

HEASARC

The HEASARC has begun a strong program of FITS[1]-related activities which forms a central component in its plans to develop an archive for data from past, present, and future high-energy astrophysics missions. Our current activities can be divided into several categories:

• developing a software interface for reading or writing FITS format data files
  
• designing and promoting FITS format standards for high energy data products
  
• converting existing archival data into FITS format
  
• distributing FITS data products on CD-ROM
  
• modifying HEASARC supported software packages to directly read or write FITS format data files
  
• developing general utility software to manipulate FITS files
  

FITSIO - a software interface for reading and writing FITS files

Since many of the HEASARC projects and activities involve dealing with data in FITS format, it is important to have a simple and efficient way to read and write FITS files. As the FITS formats have become more complicated over the years, especially with the introduction of the new binary table extension, it has become more difficult and time-consuming to write software to read or write them. So, rather than having to devote considerable effort in developing software for each particular new FITS file that came along, it was preferable in the long run to write a general subroutine interface package which could handle any type of FITS file. For this reason I wrote the FITSIO subroutine package; version 2.0 of this was publicly released in May 1991, and the latest version 3.2 was released in March 1992.

FITSIO is a machine-independent Fortran-77 subroutine interface for reading or writing data files in the FITS format. This package was written to provide an easy-to-use interface for accessing FITS files which insulates the programmer from having to deal with the complicated formatting details in the FITS file. The FITSIO package contains about 15,000 lines of ANSI-standard Fortran-77 code, most of which is machine independent; however, a small set of machine-specific subroutines are also provided for most popular computers including VAX/VMS, DECstations, Sun workstations, IBM PC's, and IBM mainframes. It supports all the commonly-used types of FITS files, including 'random grouped' data, and ASCII and binary table extensions. One of the main features of FITSIO is that it provides random access I/O capabilities which allow users to read or write information in the FITS file in any desired order. However, because of this, the package is oriented to FITS files on magnetic disk and does not support magnetic tape devices. Subroutines are provided in the FITSIO package to perform all the required types of operations on FITS files, such as: open and close files, read, write, modify, or delete header keyword values, and read or write any element of the associated data array or table. The package also contains extensive error checking to help verify that the file has a valid FITS format.

The FITSIO software is freely available from the HEASARC via either anonymous ftp or the DECNET COPY command. To copy the files using ftp, type the following command at the computer operating system prompt:

ftp tetra.gsfc.nasa.gov

or

ftp 128.183.8.77

Then type the following responses (comments appear in brackets and should not be typed on the command line):

ftp> user anonymous 
Password:  		[type your name as the password] 
ftp> cd pub/fitsio3	[to move to the fitsio (Version 3) subdirectory] 
ftp> ls			[to see a list of available files]
ftp> get read.me 	[contains latest information about FITSIO] 
ftp> get fitsio.doc	[complete user documentation for FITSIO]
ftp> get ...		[continue getting any other desired files]
ftp> quit		[close the ftp connection and exit]

NDADSA::HEASARC:[EXOSAT.XANADU.FITSIO.VERSION3]
or
15761::HEASARC:[EXOSAT.XANADU.FITSIO.VERSION3]

Any questions, bug reports, or suggested enhancements related to the FITSIO package should be sent to the author:

pence@tetra.gsfc.nasa.gov

Development of FITS Format Standards for High Energy Data Products

One of the goals of the HEASARC is to design and promote uniform format standards for different types of high energy data files in order to make it easier for both users and data analysis software to deal with data from different sources. This is an ambitious goal and it may be difficult to reach a consensus among the high energy community on standard formats. Nonetheless, the HEASARC is approaching this problem from two directions: in a top-down approach we are defining guidelines for the FITS format of various general classes of data files, such as event lists, spectra, or light curves, and in a bottom up approach we are designing formats for specific data files which adhere to and illustrate the general guidelines.

Currently we are still developing the general guidelines, and drafts of these will be circulated for comments in a future addition of this journal as well as via other available mechanisms such as the WGAS electronic mail distribution service and the alt.sci.astro.fits news group. For now, though, we can present two specific FITS file examples, one for a spectrum and one for a light curve, which are listed at the end of this article. Both of these files were generated from a observation using the Solid State Spectrometer on the Einstein (HEAO-2) Observatory. The format of the spectral file consists of a primary array containing the response matrix, followed by a binary table extension containing the observed flux, error estimates and other quantities for each detector channel. The light curve file has an empty primary array (NAXIS =0) and a binary table extension containing the observed flux rate tabulated at different times. These examples are presented here both to stimulate discussion (we would like to hear from anyone who has comments about the file formats or on our choice of keyword or column names) and to encourage others who are in the process of designing their own file formats to adopt similar formats.

Conversion of Existing Archival Data into FITS Format

The HEASARC currently has available many gigabytes of high-energy data from past X-ray and gamma-ray missions, however, almost all of these data exist in mission-specific and computer-specific formats which make it difficult for archival researchers to effectively use the data. One of the long term plans of the HEASARC is to gradually convert these data sets into FITS format to make them more accessible.

As the first step in this process, we have now completed converting the following data products into FITS format:

• all the Solid-State-Spectrometer observations taken during the lifetime of the Einstein observatory.
  
• all the Monitor Proportional Counter (MPC) observations coinciding with the SSS observations
  
• all the observations made with the Focal Plane Crystal Spectrometer (FPCS) on Einstein
  

Distribution of FITS data products on CD-ROMs

CD-ROMs provide a convenient and inexpensive medium for distributing data sets to users, and the HEASARC plans to issue many of the FITS data sets that it produces on this medium. The first CD-ROM will be available in May 1992 and will contain all the data sets described in the previous section. The contents of future CD-ROMs has not yet been determined, but new issues will be released at 6 - 12 month intervals.

Modification of Analysis Packages to Directly Read or Write FITS Format Data Files

In parallel with the above efforts to convert data files to FITS format, the HEASARC is also modifying its analysis software to be able to directly read the FITS format files. The XIMAGE image analysis and the XSPEC spectral analysis packages have already been converted. The XRONOS (timing analysis) package will be converted in the near future. Notice that this extends the use of FITS format files beyond the traditional role of merely exchanging data between different software environments. With the advent of the FITSIO software, it is now practical to directly use FITS as an internal data analysis format. One advantage of this approach is that it is only necessary to archive and maintain a single format of each file, which can then be directly read on any machine without any intermediate format conversion step.

FTOOLS - A General Package of Software to Manipulate FITS Files

As the final component of the HEASARC FITS plans, the HEASARC is undertaking a major software development project to produce a generic set of software utilities to manipulate FITS files. Each utility task will be designed to perform one simple task, such as extracting specified columns or rows from a FITS table, sorting the elements in a column, or printing the contents of a FITS file. For readers familiar with the STSDAS software package, this set of FITS utilities will be analogous to the TTOOLS set of tasks except that it will operate on FITS files instead of SDAS tables and hence it is called the FTOOLS package. This package will form the core of the HEASARC software system for reducing and analyzing data files in FITS format.

One of the main requirements on the FTOOLS software package is that it must be compatible with IRAF (the widely used data analysis package developed at NOAO) yet it must also run without any IRAF dependencies in order that the HEASARC can support both IRAF and non-IRAF users. The way this has been done is illustrated in Figure 1. First, the main body of all the FTOOLS code is written in strict ANSI standard Fortran-77 so that it is easily portable. Secondly, all the interfaces to the user (or command language) and to data files are isolated through a standard set of subroutine calls. All input and output to data files is done either to FITS files using the machine-independent FITSIO subroutine interface, or in a few cases, to simple ASCII-format text files. The interface to the user, to obtain input parameters and to return results, is based on the Fortran-77 Interface for IRAF developed at the Space Telescope Institute. As with all IRAF tasks, this interface uses an ASCII-format parameter file as the basis of all user I/O, and for interprocess communication.

The difference between making an IRAF version of the FTOOLS tasks or a stand-alone (non-IRAF) version is basically a matter of linking to the appropriate interface library. As shown on the left side of Figure 1, the FTOOLS Fortran code is linked with the IRAF Fortran-77 library to create an IRAF version. The resulting task is then 100% IRAF compatible and can be fully integrated in a script with any other IRAF task. Alternatively, to create a stand-alone version of the task, the same FTOOLS Fortran code is linked to a different interface library, as shown on the right hand side of the figure. We are currently using a prototype version of this library, which was developed at the SAO by Eric Mandel and John Roll, but expect that the full version will be available by the time this newsletter is published. The stand-alone version of the task can then simply be run from the native computer operating system prompt (one does not have to start up the IRAF CL which is needed for IRAF tasks) yet it has the same look and feel of an IRAF task in that it prompts the user for input parameters just as an IRAF task does and it provides all the usual parameter range checking and command line parsing features as in IRAF. In both cases, the tasks talk to the identical IRAF format parameter file.

The first build of the FTOOLS tasks was recently completed, and it successfully demonstrated that the tasks could be run in both the IRAF and the non-IRAF environments. Consequently, this same software development approach is now being adopted for other projects within the HEASARC. Work on more of the FTOOLS tasks is continuing, and a first release of the FTOOLS software will be made in mid 1992.

figure 1

FITS file example: SSS spectrum header

 SIMPLE  =                    T / file does conform to FITS standard
 BITPIX  =                  -32 / number of bits per data pixel
 NAXIS   =                    0 / number of data axes
 EXTEND  =                    T / FITS dataset may contain extensions
 
 CONTENT = 'SPECTRUM'           / file contains spectrum
 DATE    = '20/12/91'           / date this FITS file was created (dd/mm/yy)
 ORIGIN  = 'HEASARC/GSFC'       / organization which created this file
 TELESCOP= 'EINSTEIN'           / also known as HEAO-2
 INSTRUME= 'SSS     '           / Solid State Spectrometer
 OBJECT  = 'X0535+262'          / observed object
 RA      =           83.9458313 / right acension of target (decimal degrees)
 DEC     =           26.2886105 / declination of the target (decimal degrees)
 EQUINOX =               1950.0 / equinox of celestial coord. system
 DATE-OBS= '25/03/79'           / date observations were made (dd/mm/yy)
 TIME-OBS= '17:38:16'           / time observations were made (hh:mm:ss)
 DATE-END= '25/03/79'           / date observations ended (dd/mm/yy)
 TIME-END= '19:46:34'           / time observations ended (hh:mm:ss)

 END
 ==========================================
 *** No Primary Array ***
 ==========================================
 XTENSION= 'BINTABLE'           / binary table extension
 BITPIX  =                    8 / 8-bit bytes
 NAXIS   =                    2 / 2-dimensional binary table
 NAXIS1  =                   12 / width of table in bytes
 NAXIS2  =                  128 / number of rows in table
 PCOUNT  =                    0 / size of special data area
 GCOUNT  =                    1 / one data group (required keyword)
 TFIELDS =                    4 / number of fields in each row
 TTYPE1  = 'CHANNEL '           / detector channel number
 TFORM1  = 'I       '           / data format of the field: 2-byte INTEGER
 TTYPE2  = 'RATE    '           / observed event count rate
 TFORM2  = 'E       '           / data format of the field: 4-byte REAL
 TUNIT2  = 'COUNTS/S'           / physical unit of field
 TTYPE3  = 'STAT_ERR_RATE'      / statistical rms error on the count rate
 TFORM3  = 'E       '           / data format of the field: 4-byte REAL
 TUNIT3  = 'COUNTS/S'           / physical unit of field
 TTYPE4  = 'DQF     '           / data quality flag
 TFORM4  = 'I       '           / data format of the field: 2-byte INTEGER
 EXTNAME = 'SPECTRUM'           / name of this binary table extension
 TELESCOP= 'EINSTEIN'           / also known as HEAO-2
 INSTRUME= 'SSS     '           / Solid State Spectrometer
 OBJECT  = 'X0535+262'          / observed object
 RA      =           83.9458313 / right acension of target (decimal degrees)
 DEC     =           26.2886105 / declination of the target (decimal degrees)
 EQUINOX =               1950.0 / equinox of celestial coord. system
 DATE-OBS= '25/03/79'           / date observations were made (dd/mm/yy)
 TIME-OBS= '17:38:16'           / time observations were made (hh:mm:ss)
 DATE-END= '25/03/79'           / date observations ended (dd/mm/yy)
 TIME-END= '19:46:34'           / time observations ended (hh:mm:ss)
 STRTTIME=            -24301304 / obs. start time (seconds from 01/01/80)
 STOPTIME=            -24293606 / obs. stop time (seconds from 01/01/80)
 EXPOSURE=         3358.7199707 / integration time (seconds)
 MEANEPOC=          449.7778320 / mean obs time (decimal day of year 1978)
 MAJORFRM=                 1830 / major frame start time
 MINORFRM=                  120 / minor frame start time
 NEXPOSE =                    5 / total number of exposures
 EXPOS1  =         1761.2799072 / length of exposure (seconds)
 EXPOS2  =         1597.4399414 / length of exposure (seconds)
 EXPOS3  =            0.0000000 / length of exposure (seconds)
 EXPOS4  =            0.0000000 / length of exposure (seconds)
 EXPOS5  =            0.0000000 / length of exposure (seconds)
 MEPOC1  =          449.7460938 / mean epoch of exposure (day of 1978)
 MEPOC2  =          449.8129883 / mean epoch of exposure (day of 1978)
 MEPOC3  =          449.0000000 / mean epoch of exposure (day of 1978)
 MEPOC4  =          449.0000000 / mean epoch of exposure (day of 1978)
 MEPOC5  =          449.0000000 / mean epoch of exposure (day of 1978)
 EFFAREA =               180.00 / on-source detector area (cm**2)
 BACKSCAL=                1.000 / background scale factor
 CORRSCAL=                0.000 / correction scale factor
 JCGPARM =            110.00000 / background prediction parameter
 DEADTIME=            4030000.0 / dead time parameter
 MEAN_ICE=                 1.40 / mean water ice coating on detector
 BACKFILE= 'a053526.bk'         / background FITS file for this object
 RESPFILE= 'a053526.rs'         / response FITS file for this object
 CORRFILE= 'a053526.cr'         / correction FITS file for this object
 
 COMMENT   This table summarizes an X-ray spectrum observed with the Solid
 COMMENT   State Spectrometer on the EINSTEIN (HEAO-2) satellite. For each
 COMMENT   of the 128 detector channels the table lists the following values:
 COMMENT
 COMMENT   CHANNEL       detector channel number (1-128)
 COMMENT   RATE          observed photon count rate before background subtraction
 COMMENT   STAT_ERR_RATE statistical rms error of the count RATE
 COMMENT   DQF           data quality flag (0=bad, 1=good)
 COMMENT
 COMMENT   The first 14 channels and the last 20 channels out of the 128 SSS
 COMMENT   detector channels did not produce any useful data, therefore the
 COMMENT   corresponding quantities for these channels are all set equal to
 COMMENT   an IEEE NaN value (all bits set to 1).
 
 HISTORY   Input PHA spectrum file = SA053526.PHA
 END



	FITS file example:  SSS light curve header

 SIMPLE  =                    T / file does conform to FITS standard
 BITPIX  =                    8 / number of bits per data pixel
 NAXIS   =                    0 / number of data axes
 EXTEND  =                    T / FITS dataset may contain extensions
 CONTENT = 'RATEFILE'           / file contains rate data
 ORIGIN  = 'HEASARC/GSFC'       / High Energy Astrophys. Sci. Arch. Res. Center
 DATE    = '19/12/91'           / file creation date (dd/mm/yy)
 TELESCOP= 'Einstein'           / mission name (also known as HEAO-2)
 INSTRUME= 'SSS     '           / Solid-State Spectrometer
 OBJECT  = 'X0535+262'          / name of observed object
 RA      =            83.945831 / R.A. of object (deg)
 DEC     =            26.288610 / Declination of object (deg)
 EQUINOX =               1950.0 / equinox of celestial coord. system
 DATE-OBS= '25/03/79'           / UT date of sequence start (DD/MM/YY)
 TIME-OBS= '17:38:18'           / UT time of sequence start (hh:mm:ss)
 DATE-END= '25/03/79'           / UT date of sequence end (DD/MM/YY)
 TIME-END= '19:46:39'           / UT time of sequence end (hh:mm:ss)
 END
 ==========================================
 *** No Primary Array ***
 ==========================================
 XTENSION= 'BINTABLE'           / binary table extension
 BITPIX  =                    8 / 8-bit bytes
 NAXIS   =                    2 / 2-dimensional binary table
 NAXIS1  =                   10 / width of table in bytes
 NAXIS2  =                   48 / number of rows in table
 PCOUNT  =                    0 / size of special data area
 GCOUNT  =                    1 / one data group (required keyword)
 TFIELDS =                    3 / number of fields in each row
 TTYPE1  = 'TIME    '           / time of measurement
 TFORM1  = 'I       '           / data format of field 1: 2-byte INTEGER
 TUNIT1  = 's       '           / physical unit of field
 TTYPE2  = 'RATE    '           / observed event count rate
 TFORM2  = 'E       '           / data format of field 2: 4-byte REAL
 TUNIT2  = 'counts/s'           / physical unit of field
 TTYPE3  = 'STAT_ERR_RATE'      / statistical rms error on the count rate
 TFORM3  = 'E       '           / data format of field 3: 4-byte REAL
 TUNIT3  = 'counts/s'           / physical unit of field
 EXTNAME = 'RATEFILE'           / name of this binary table extension
 TSCAL1  =   8.191998291016E+01 / scaling factor for column 1
 TZERO1  =  -2.430134295999E+07 / scaling zero point for column 1
 DATEZERO= '01/01/80'           / UT date zero point of TIME column (DD/MM/YY)
 TIMEZERO= '00:00:00'           / UT time zero point of TIME column (hh:mm:ss)
 BARYTIME=                    F / barycentric correction not applied to times
 DEADCORR=                    F / detector deadtime correction not applied
 TELESCOP= 'Einstein'           / mission name (also known as HEAO-2)
 INSTRUME= 'SSS     '           / Solid-State Spectrometer
 OBJECT  = 'X0535+262'          / name of observed object
 RA      =            83.945831 / R.A. of object (deg)
 DEC     =            26.288610 / Declination of object (deg)
 EQUINOX =               1950.0 / equinox of celestial coord. system
 DELTIME =   8.191998291016E+01 / time interval between measurements (seconds)
 RATEMEAN=               3.2740 / mean count rate of the observation (counts/s)
 RATERMS =               0.0290 / RMS error of RATEMEAN (counts/s)
 DATE-OBS= '25/03/79'           / UT date of sequence start (DD/MM/YY)
 TIME-OBS= '17:38:18'           / UT time of sequence start (hh:mm:ss)
 DATE-END= '25/03/79'           / UT date of sequence end (DD/MM/YY)
 TIME-END= '19:46:39'           / UT time of sequence end (hh:mm:ss)
 DATEDEFR= '23/03/79'           / UT date of last instrument defrost (DD/MM/YY)
 TIMEDEFR= '17:16:48'           / UT time of last instrument defrost (hh:mm:ss)
 RATEPARM=                 1523 / Rate parameter value
 OPTORATE=                2.000 / Opto Rate parameter value
 MINCHAN =                   20 / minimum included detector energy (PHA) channel
 MAXCHAN =                   75 / maximum included detector energy (PHA) channel
 COMMENT
 COMMENT  This table contains a time series of X-ray flux measurements
 COMMENT  derived from observations taken with the Einstein Solid State
 COMMENT  Spectrometer.  The first column defines the midpoint in time of
 COMMENT  the integration bin, the second column gives the measured flux
 COMMENT  rate in counts per second, and the third column gives the
 COMMENT  statistical RMS error on the count rate. There may be gaps in
 COMMENT  the data due to earth occultations or other reasons.  The times
 COMMENT  given here refer to the photon arrival time at the spacecraft;
 COMMENT  no barycentric correction has been applied.
 COMMENT
 COMMENT  Note that the TIME column contains Integer*2 values which
 COMMENT  represent the sequential bin number of the measurement.
 COMMENT  The midpoint in time of the corresponding measurement in seconds
 COMMENT  relative to January 1, 1980 (the zero point is defined by the
 COMMENT  DATEZERO and TIMEZERO keywords) is given by scaling the
 COMMENT  bin number by the TSCAL1 and TZERO1 keyword values, i.e.,
 COMMENT     Time (in seconds) = bin number * TSCAL1 + TZERO1
 COMMENT
 HISTORY   Input RBF rate file = SA053526.RBF;2
 END

Proceed to the next article

Return to the previous article

Select another article