The HEASARC RXTE Data Archive

A. Smale
RXTE Guest Observer Facility

Abstract

The Rossi X-ray Timing Explorer (RXTE) Archive contains a large and growing amount of high-quality timing and broad-band spectral data from a variety of X-ray sources. This article provides an introduction to the Proportional Counter Array (PCA), High-Energy X-ray Timing Experiment (HEXTE), and All-Sky Monitor (ASM) instruments on board RXTE, and a userguide to the Archive. We describe the data structure; provide step-by-step instructions on how to browse and retrieve RXTE data using W3BROWSE, anonymous FTP, or XDF; and give pointers for data analysis.

1. Introduction: The Rossi X-ray Timing Explorer

The RXTE satellite was launched on December 30, 1995 with the primary objective of studying the structure and dynamics of compact X-ray sources, including neutron star and black hole systems both within and beyond our Galaxy. RXTE was designed to study the intensity variations of these objects over timescales from microseconds to years. Significant discoveries have already been made using RXTE data on both ends of this range of timescales, including the discoveries of millisecond pulsations in X-ray bursts, kiloHertz quasiperiodic oscillations (QPOs) in low-mass X-ray binaries, and various flavors of long-term variability in LMXBs and quasars. In addition to the advanced timing capabilities that give the satellite its name, RXTE has two other critical strengths of great importance to X-ray and multiwavelength astrophysics: its broad-band energy response, and its unequaled observing flexibility.

RXTE carries three scientific instruments. The PCA covers the 2-90 keV energy range, and consists of five identical collimated multi-anode Proportional Counter Units (PCUs). The total effective area of the PCA at the peak of its efficiency curve is approximately 7000 cm², making it the largest proportional counter yet flown. The detectors have an 18% energy resolution at 6 keV and 255-channel pulse-height discrimination. The PCA was built at NASA/GSFC.

The HEXTE consists of two independent clusters of detectors each containing four NaI/CsI phoswich scintillation counters, covering the energy range 15-250 keV with an intrinsic spectral resolution better than 18% at 60 keV. The two clusters contain mutually orthogonal rocking mechanisms, which can be moved independently to provide near-simultaneous measurements of the internal and cosmic X-ray background up to 3 degrees on either side of the source. The PCA and the HEXTE are co-aligned with fields of view of 1 degree. The HEXTE was designed and built at the Center for Astrophysics & Space Sciences at the University of San Diego, California.

The ASM alerts scientists to flares and changes of state in X-ray sources, and produces long-term intensity histories of bright X-ray sources. It consists of three rotating Scanning Shadow Cameras that can scan about 80% of the sky every 90 minutes. The cameras can also provide positions accurate to ~3' for bright transients. The cameras are sensitive in the 2-10 keV range, and were built at MIT. The data from the PCA and ASM pass through the Experiment Data System (EDS, also built at MIT) for preliminary processing.

The PCA and HEXTE data from the one-month In-orbit Checkout (IOC) phase in January 1996 have already been made public in the RXTE Archive (~8 GB of data), along with the pointed data from some non-proprietary Target of Opportunity programs (24 GB at the time of writing), and the slews to and from all targets, which are also non-proprietary (24 GB). The ASM data are made publicly available as soon as they are processed. The above-listed datasets have already been transferred to the RXTE Archive at the HEASARC.

In addition to these holdings, in March 1998 data began entering the public domain from observations performed under the RXTE Guest Observer program. The standard proprietary period for RXTE data obtained in response to accepted GO proposals is twelve months from the date of receipt of the data by the PI. However, due to reprocessing activities, the proprietary period for AO-1 data taken before December 16, 1996 has been extended by NASA Headquarters until six months after the receipt of the reprocessed data tape by the PI. By the end of 1998, a total of 120 GB of GO data will have been made public. After all the data from AO-1 through AO-3 have turned public, the grand total will exceed 300 GB.

In Section 2 of this article we describe the types of data available from the PCA and HEXTE instruments, and the structure of these data in the archive. Section 3 contains a primer on how to search the databases and retrieve datasets of interest. Data analysis tools and documentation are also described. In Section 4, we provide information about the available resources for obtaining, interpreting, and analyzing data from the All-Sky Monitor.

2. RXTE Data

2.1 What types of data are available?

The data from the PCA can be binned and telemetered in six different modes simultaneously by the independent Event Analyzers (EAs) in the EDS system. Two of these EAs are reserved for two standard PCA modes, with timing and spectral parameters that will remain unchanged throughout the mission to provide a uniform mission data bank. The two modes are a time series mode, Standard 1, with 0.125-sec temporal resolution and no energy resolution, and a spectral mode, Standard 2, with 128 channels of spectral information and a 16-second temporal resolution. Users typically begin their analysis by examining the Standard 2 data to gain an overview of the observation; in fact, for many applications the Standard 2 data may be sufficient for the user's needs.

The other four available EDS configurations were chosen by the Principal Investigator of the original observation, generally with the goal of maximizing the temporal and spectral information obtained by the PCA, while staying within the constraints of the permitted telemetry bandwidth. For fainter sources, the "Good_Xenon" modes are usually chosen. Good Xenon data use up two of the four available EAs, and when combined into an event list, provide the full 256-channel resolution of the PCA instrument for all events that survive background rejection, with event time-tagging accurate to 0.95 microseconds. Each event is also tagged with detector and layer information, enabling the most optimal background subtraction methods.

For brighter sources, data may have been obtained in a binned mode, a single-bit mode (for very high time resolution data), an event mode, or some combination of these. For full details on exactly how each of the possible modes is defined we refer the user to the complete userguide on the RXTE GOF Website, or to the Technical Appendix to the Announcements of Opportunity for observing time. However, one important fact to be absolutely clear on is that these data streams are received in parallel -- the same events can be telemetered to ground in up to six different ways, depending upon the configurations chosen by the proposer in consultation with the GOF or SOF.

The HEXTE has a similarly large range of possible configurations, including an Archival mode that runs continuously, accompanied by a user-selected mode. The vast majority of HEXTE observations are now performed in the mode known as E_8us_256_DX1f, which is an event mode with 256-channel resolution and an 8 microsecond time resolution.

The RXTE Archive currently contains data products for each observation. These are: a PCA light curve and spectrum from Standard 2 data, a HEXTE light curve and spectrum, and a filter file containing a compendium of collected and derived parameters of importance for screening, background subtraction and data analysis (pointing position, elevation angle, number of PCUs on, etc). The light curves and spectra were derived automatically with a very rudimentary screening algorithm, and no background subtraction. This renders them of some illustrative value for bright sources, and of very little use for fainter targets. Once the Archive matures and becomes established, a high priority for future work is the creation of more advanced and reliable data products as a convenience to archival users.

2.2 How are the data organized in the Archive?

2.2.1 Directories

Archival RXTE data occupy the xte/data/archive directory on the anonymous FTP server at heasarc.gsfc.nasa.gov. Directories at the next level down are named AO0 for IOC data, AO1 for AO-1 data, etc. Note that you will see directories for data that are not yet public. The links are there, but the data files themselves will not be accessible to you until the proprietary period expires.

Below this level, RXTE data files are arranged in a hierarchical set of directories identical in structure to the arrangement of files on a GO's data tape. First come directories named for the proposal number (e.g. "P10066"); the next level contains the individual observations ("ObsID's"), while the final level contains directories for each of the Subsystems (PCA, HEXTE, ACS etc.). In the catalogue of publicly available RXTE data, individual entries correspond to the ObsID level.

2.2.2 Observation IDs

Since the ObsID is the link from the browseable catalogue to the data themselves, it is worth explaining its nomenclature. Each ObsID corresponds to a single observation, where "observation" refers to a temporally contiguous collection of data from a single pointing. The format for the ObsIDs is as follows:

NNNNN-TT-VV-SSX

1. NNNNN is the five-digit proposal number assigned by the GOF, identical to that contained in the name of the

parent directory.

2. TT is the two-digit target number assigned by the GOF. Note that for the case of only one target, the target

number may be zero.

3. VV is the two-digit viewing number, assigned by GOF, which tracks the number of scheduled looks at the

target. In particular the viewing number corresponds to:

• different (requested) observations of the same target (e.g., at different epochs for monitoring),

• different instrument configurations during the same pointing,

• different scans for scan-mapping of extended sources.

4. SS is the two-digit sequence number used for identifying different pointings that make up the same viewing if the SOF decided, for operational reasons, to split that viewing into more than one chunk or if it was broken up by e.g. a TOO observation.

5. X, the (optional) 15th character, indicates:

     A       Slew before observation
     Z       Slew after observation
     S       Raster scan observation
     R       Raster grid observation
     0-9     Segments of a long observation (i.e., > 8 hours)
     b-r     (reserved for) Real-time configuration changes

When not present, it indicates that the data correspond to a regular pointed observation (< 8 hrs), or the last segment of a longer observation.

2.2.3 Spacecraft Subsystems

Below each ObsID directory are a set of 15 subdirectories, each containing data files derived from a single spacecraft Subsystem. The corresponding directory names are as follows:

ace - Attitude Control Electronics & star trackers

acs - Attitude Control System

cal - References to files in the Calibration Database

clock - Time delta correction data from Mission Operations Center

eds - Experiment Data System housekeeping

fds - Flight Data System

gsace - Gimbals and Solar Array Control Electronics

hexte - HEXTE science and housekeeping data

ifog - Interferometric Fibre Optics Gyroscope

ipsdu - Instrument Power Switching and Distribution Unit

orbit - Orbit ephemeris from the Flight Dynamics Facility

pca - PCA science and housekeeping data

pse - Power System Electronics

spsdu - Spacecraft Power Switching and Distribution Unit

stdprod - Standard products generated by the XSDC

Although they represent the lowest rung in the directory hierarchy, the Subsystem directories do not necessarily contain files of one type. In most cases, a further division is made based on Application, the term used for a distinct source of telemetry. In the case of the PCA, the six Event Analyzers are considered applications.

Also in the ObsID directory are various index files (FI*) used by XDF (the XTE Data Finder), the XTE filter tools, and other software to navigate the underlying data files.

2.2.4 Example of Directory Structure

Within W3BROWSE, the relevant database to search for RXTE pointed (PCA and HEXTE) data in the public domain is XTEPUBLIC. An example will clarify how XTEPUBLIC relates to the ObsID and manifold directory structure. Searching XTEPUBLIC for observations of 4U1538-52 yields the ObsIDs:

10145-01-01-00 4U_1538-52

10145-01-01-01 4U_1538-52

10145-01-02-01 4U_1538-52

10145-01-02-02 4U_1538-52

10145-01-02-03 4U_1538-52

...

10145-01-01-00A 4U_1538-52_Slew

10145-01-01-00Z 4U_1538-52_Slew

...

The first few ObsIDs contain data from a series of observations of the source. The last two ObsIDs listed above contain slew data from before and after the first observation. (Other ObsIDs will appear from other observations of the same source; if these are limited to slew files, the data from the pointed observations are not yet public.)

For the user of anonymous FTP, the archival directory containing PCA data from the first ObsID (the pointed observation) would be seen as:

xte/data/archive/AO1/P10145/10145-01-01-00/pca

Note that although you can work your way down the directory structure using FTP, it is often more convenient to use W3Browse to find and retrieve the data, or retrieve a whole ObsID using FTP.

3. Browsing, retrieving and analyzing PCA and HEXTE data

3.1 When will the GO data I'm interested in go public?

The RXTE GOF has developed the Public Data Web tool (http://heasarc.gsfc.nasa.gov/docs/xte/xte_public.html ) to allow users to find out when proprietary data will go public. Users can search on target name, PI, proposal number, complete ObsID, and/or date. For example, to find out what data went public in March 1998, one would enter in the appropriate box "Mar-*-1998". Note that if the data have not yet been (re)processed by XSDC, the go-public date will not be known and the observations will not appear in the output from the Public Data Web tool.

The user may also search the XTEPUBLIC catalogue using the HEASARC's W3BROWSE utility (http://heasarc.gsfc.nasa.gov/W3Browse/) to find out which datasets are currently available.

3.2 How do I find and retrieve the data I want?

There three methods for retrieving public data: the web-based W3Browse, anonymous FTP, and the XTE Data Finder (XDF).

3.2.1 Using W3Browse

W3Browse allows you to browse available HEASARC datasets via the Web. Link to the main W3Browse page, and choose the "Advanced" version (to allow greater search flexibility). In the "Advanced" interface, check the RXTE mission box, choose to search either by object name/coordinates or by parameters, and then begin your search:

a. Choose the XTEPUBLIC catalogue.

b. Enter search criteria into appropriate boxes. If you chose to search by name, you might enter e.g. 4U1907+09 into the Object Name or coordinates box, and click Submit. To search by parameters, you'll click again to begin your search, and perhaps choose data by proposal number (called PRNB within W3BROWSE), ObsID etc. Then click Submit.

c. A successful search will display the corresponding catalogue entries, i.e. the ObsIDs, one per line. Check the box(es) to the left of the ObsID(s) you're interested in. (Remember that ObsIDs ending in an A or a Z are slews. In most cases, the target name should also indicate if the data correspond to a slew.)

d. Below this listing, you'll find a choice of datasets. You will probably want one of the FULL datasets, e.g. FULL RAW OBSID or FULL RAW PROPOSAL. (The REDUCED OBSID DATA contains standard products for each observation which may be useful as an indication of the strength of the source and the duration and coverage of the observation, but these are not intended for use in any detailed scientific analysis.) Choose one by clicking on it to highlight it.

e. Then click on the radio button to "Retrieve data products in selected categories for selected observations." Note that the default setting, "List all data products for selected observations" is not a helpful option for RXTE datasets, which can contain many hundreds of files - unless you know you only need one or two.

f. Click on Submit. W3BROWSE will now construct the appropriate tar command, and tar up the data you have requested. This may take a while.

g. W3BROWSE will bring up a new page when your tar job is complete. You may now click on "Download TAR file" to initiate the transfer of the tar file to your host machine. Make sure you have enough disk space! - the total size of the tar file will be indicated.

h. When untarred, your data files and index files will appear in the usual configuration for RXTE datasets, as described above.

3.2.2 Anonymous FTP

Anonymous FTP can be a convenient alternative to W3BROWSE, if you don't wish to tie up your Web browser for the duration of the data retrieval. Here's an example of the steps you might follow:

1. "ftp heasarc.gsfc.nasa.gov", giving 'anonymous' as your name, and your Email address as the password.

2. "cd xte/data/archive" (stopping to read the information presented here).

3. "cd AO1". "get P10145.tar" to obtain all the public data from this proposal. The FTP server will automatically create the tarfile on your home machine in the directory from which you initiated the FTP run - make sure you have sufficient space. You might want to "cd P10145" and obtain a subset of the data (e.g. "get 10145-02-02-0.tar").

There is no advantage to pulling over a gzipped file using e.g. "get 10145-02-02-00.tar.gz". All the datafiles within the directory structure are already gzipped, and the savings in transfer time will be minimal or non-existent.

If you try to FTP data which is still proprietary, you'll end up with a tarfile containing clock and orbit information, calibration files and index files, but no science data.

3.2.3 XDF

The XTE Data Finder (XDF) is a Graphical User Interface that may be used to search for data and retrieve them from the HEASARC archive. One advantage of using XDF is that it allows you complete access to information about the data modes used for each observation.

1. Obtain the Top-Level Master Index File (FMI), using W3BROWSE. (To do this, select any random RXTE observation using steps 1 and 2 of the recipe above in 3.2.1, then choose "XTE Top-Level FMI" as the category of data product to retrieve.)

2. Once you have this FMI on your home machine, invoke XDF.

3. Change the default FTP setting in the top left hand corner of the GUI from "FTP No Files" to "FTP Any Files", and enter the path of the directory where you're keeping the FMI into the "Path:" box.

4. Click on the "Make ObsList" button. While this button stays lighter than the surrounding GUI, XDF is hard at work reading the FMI.

5. You can now scroll up and down through the entire available archive in the "Observations" window, or make selections by source and/or date in the "Sources" or "Time Ranges" window. (To enter a list of sources, click on the "Edit" button.)

6. Once you've chosen the sources or data of interest, click on "Make AppIDConfigList", choose the instruments and data configurations of interest, and click on "Make Filelist".

During this stage, XDF will automatically connect to the HEASARC archive and download any necessary index files, or any data files that you request. The progress of these FTP sessions will be logged in a separate window. Data will be saved on your current disk in the familiar RXTE directory structure.

3.3 Cautions and Caveats

RXTE data will be released into the archive on a weekly basis. Observations newly turning public before 12:01a.m. on a given Saturday will be available (i) the following Monday, for anonymous FTP access; (ii) Tuesday, for W3BROWSE access.

RXTE data will go public on an ObsID basis, based on the date when the tape containing that ObsID was mailed to the original PI. Thus, not all the parts of a given proposal will necessarily go public at the same time, particularly in the case of large proposals or long-term monitoring campaigns.

RXTE datasets may be larger than you are accustomed to, and response may be slow at peak times, particularly if multiple users are attempting to get their hands on the same dataset. Please ensure you have sufficient disk space for the transfers, and try to plan your data retrieval to avoid times of heavy network traffic. If you have problems, try again later.

3.4 Analyzing PCA and HEXTE data

A full description of the extensive suite of tools available for RXTE data analysis is beyond the scope of this article. Complete documentation for the installation and use of all tools is available online from the GOF homepage by following the link to "Data Analysis & Processing." Here are some highlights:

Selective untarring of RXTE data can be performed with the XSUT tool, which is equipped with a graphical user interface (GUI) and is available for the first time under FTOOLS 4.1.

Finding your way through the hierarchical directory structure to identify the data you want to reduce is made easy using XDF. Its end-product is a list of filenames corresponding to the data you want to analyze, as chosen by target name, time of observation, instrument, and configuration.

Data manipulation and filtering, and the extraction of spectra and lightcurves, can be performed using the XTE (and other) FTOOLs. You can use the FLAUNCH GUI to run these tools and keep a log of your activities.

The individual instrument teams play the major role in calibrating their instruments. The GOF provides a repository for calibration files and instrument-team-provided software, along with information on their application. The teams provide documentation on their calibration efforts on their own Web pages. There are extensive links to this information via the GOF "About RXTE" and "Data Analysis" pages.

As for all HEASARC X-ray datasets, XSPEC and XRONOS are available to perform spectral and timing analysis.

Comprehensive instructions for reducing RXTE data are provided in three complementary online guides:

(i) "The RXTE Getting Started Guide" contains information about how to read your RXTE data tape, the directory structure on your tape, how to use XDF to browse your files, and how to install all the necessary software.

(ii) "The ABC of XTE" provides extensive information about RXTE data files, data screening and filtering, the extraction of spectra and light curves, background subtraction, etc.

(iii) "The RXTE Cook Book" provides a large number of step-by-step recipes for the most commonly-performed RXTE data reduction tasks. Complete end-to-end recipes are available for the reduction of PCA, HEXTE and ASM data, along with individual recipes for creating filter files, working with realtime data, screening and selection of data, correcting for background and deadtime, creating response matrices, performing Fourier analysis of PCA data and searching for high-frequency QPO, performing pulse phase spectroscopy, managing a FITS database, etc.

RXTE data analysts may send queries on any aspect of RXTE to xtehelp@athena.gsfc.nasa.gov. However, the FAQ page ( http://heasarc.gsfc.nasa.gov/docs/xte/ftools/xtefaq.html) should be consulted first, to see if it contains the information required.

4. ASM data

4.1 Overview of the ASM Data Products

The ASM Products Database makes available results from the RXTE All Sky Monitor. These results include light curves for around 300 sources, intensities in three sub-energy bands (or "colors") for each of these sources, and a history of the ASM pointings. The database provides these data files in FITS format. (MIT's ASM Light Curves Overview page provides GIF images and ASCII files for the light curves).

The database consists of both Quicklook and Definitive versions of the Products. The Quicklook Products are derived from analysis of the real-time data performed in the RXTE SOF. They are normally updated every three hours and include data from up to the previous two weeks. The Quicklook Products are superceded by updates to the Definitive Products, which are produced at MIT from the final production data. The Definitive Products are usually updated weekly.

An important feature of the light curves and color files is that because two of the ASM Scanning Shadow Cameras (SSCs) have overlapping fields of view and because data from each camera are analyzed independently, it is often the case that sources may have multiple intensity measurements at the same time. These measurements are, however, independent and may be combined as such.

Quicklook ASM results are also available via the ASM Weather Map. The map provides an "at a glance" look at the X-ray sky, and an accompanying table provides the latest x-ray intensity values for over one hundred active sources in the ASM catalogue. Selecting a source name from the table produces a plot and ASCII table of the light curve over the previous two weeks.

4.2 Access to the ASM Data Products

4.2.1 Via FTP

The ASM Data Products are accessible from the ASM Data Products Page. Using direct anonymous FTP to heasarc.gsfc.nasa.gov, the Products may be found in the following subdirectories of ftp://heasarc.gsfc.nasa.gov/xte/data/archive/ASMProducts:

The Definitive Products:

Light curves: definitive_1dwell/lightcurves

Colors: definitive_1dwell/colors

Pointings: definitive_1dwell/supplemental/pointings

The Quicklook Products:

Light curves: realtime_current/lightcurves

Colors: realtime_current/colors

Pointings: realtime_current/supplemental/pointings

4.2.2 Via W3Browse

For the Definitive Products:

Choose the "XTE All-Sky Monitor Long-term Observed Sources" (XTEASMLONG) catalog. After giving the name or coordinates of a particular source, the user may then either choose to retrieve the products individually or in data product sets. The STANDARD set consists of both the light curve and the color file for the object. The POINTINGS set consists of all of the weekly ASM pointing files.

For the Quicklook Products:

Choose the "XTE All-Sky Monitor Quicklook Observed Targets" (XTEASMQUICK) catalog. After giving the name or coordinates of a particular source, W3Browse will list the intensity values at times spanning the previous two weeks. After choosing any of these rows, the user may then choose to retrieve the quicklook products either individually or in the data product sets. As with the definitive products, the STANDARD set consists of both the light curve and the color file for the object. The POINTINGS set consists of all the daily ASM pointing files presently in the real-time archive.

4.3 Analysis of ASM Products

The ASM Data Products can be analyzed using the FTOOLS software. See the "Working with the ASM Data Products" recipe in the RXTE Cook Book for details.

Acknowledgments

The construction, population and maintenance of the RXTE Archive is a collaborative effort between several groups at NASA/GSFC. The XTE Science Data Center (XSDC) perform the pipeline processing to convert the production data into FITS format and distribute data to GOs. They provide these data to the GOF, along with the distribution logs stating when each dataset should go public. The XSDC pipeline is based around the core dmIngest and XFF (XTE Fits Formatter) software provided by the SOF and GOF respectively. The GOF is responsible for data transfer and population of the Archive and some quality control functions. HEASARC handles the database tables for the W3BROWSE online interface, and maintains the Archive. While a rather large cast of characters contributed to this scheme, we name here only the principal architects: Arnold Rots, Randy Barnette, Bob Patterer, Mike Tripicco, and Ed Sabol.

References

The RXTE GOF homepage: http://heasarc.gsfc.nasa.gov/docs/xte/

The HEASARC homepage: http://heasarc.gsfc.nasa.gov/

The Technical Appendix to the NASA Research Announcements; can be obtained from ftp://heasarc.gsfc.nasa.gov:/xte/nra/appendix_f

The ASM Data Products page: http://heasarc.gsfc.nasa.gov/docs/xte/asm_products.html

The PCA Team homepage: http://heasarc.gsfc.nasa.gov/docs/xray/xte/pca/

The HEXTE Team homepage: http://mamacass.ucsd.edu:8080/hexte/hexte.html

The ASM Team homepage: http://space.mit.edu/XTE/XTE.html

(All of the above are accessible via the GOF pages.)