National Aeronautics and Space Administration

Rex is a script to make basic RXTE PCA and HEXTE data analysis more straightforward.

• About Rex
• Using Rex
• Trouble shooting

About Rex

General description:

Rex is a script designed to run through the basic data reduction of PCA and HEXTE data for multiple observations within a given proposal. It reads the index files in the RXTE database structure and extracts the most commonly used data products from the PCA Standard2 or the HEXTE Archive data for as many ObsIds as it finds in the database. In addition to filtering the data using the GOF-recommended criteria, the script produces model background files to subtract from the data. The final products are a light curve and a spectrum for each individual observation, plus an overall light curve and spectrum for the all observations combined.

Note: Rex is meant to be a convenient tool for doing the same basic analysis on a large amount of data, not as a one-script-fits-all substitute for understanding how to use XTE data and software. Before using Rex, familiarize yourself with your data and the software tools that Rex calls by reading the ABC Guide and by analyzing at least one observation by hand, start to finish, following the end-to-end PCA and HEXTE recipes.

Rex was designed to be easy to use and fairly robust, in some ways at the expense of flexibility. The script can be run from the command line with nothing more specified than the proposal level directory of the data to be extracted. Its defaults are intended for the analysis of faint sources such as AGN, and its power lies in its convenience for large numbers of observations, especially in the case of long monitoring campaigns. Most of the defaults, however, can be changed allowing the scientist to customize the analysis without having to do it all by hand.

What it does:

The user obtains an auxiliary directory of tools (available as a small tar file on our archive), places it in the working directory, and runs the script as outlined below. Rex uses the contents of that "aux" directory to perform the following steps for each ObsId found in the proposal's Master Index file:

1. check that it is not a slew (unless flagged -s to extract the slew data) and is the right target (if one is specified);
2. run xtefilt to produce an updated filter file containing the quantities found in aux/appidlist; this filter file will be put in the proposal/ObsId/stdprod/ directory, named with the suffix ".xfl", and listed in the standard products index file (the FIST... file) for that ObsId;
3. make a directory for the ObsId in both the aux directory (for file lists, GTIs, background files, etc.) and in the working directory (for resulting light curves and spectra); compile a list of Standard2 files by dumping FIPC index file;
4. run pcabackest to produce modeled background data for each Standard 2 file found in the ObsId/pca/ directory's index file (the FIPC... file); use the background models listed in aux/model.files and the SAA history file there; the output background files go into the aux/ObsId directory along with a list of all files successfully created;
5. run maketime to produce a list of Good Time Intervals (GTI) to be extracted; it uses the filter file and applies the criteria in aux/expression.txt; the result is checked for good time, and if no time fits those criteria, Rex goes onto the next ObsId without attempting to extract;
6. run saextrct to produce spectra and/or lightcurves for source and background files (assuming GTI and background files were created successfully) using the defaults below;
7. subtract the background light curve from the source using lcmath and/or writes the background spectrum into the BACKFILE keyword of the source spectrum;

Finally, Rex creates overall lightcurves and spectra for all observations combined.

If the user is analyzing HEXTE data, the appropriate background is assembled from the off-source pointings, and subtracted from the data as per our online HEXTE recipe.

What it makes:

The default extracted products are light curves and spectra extracted from the PCA Standard 2 data using:

• layer 1 only, for better signal to noise;
• PCUs 0, 1, and 2, since PCUs 3 and 4 go on and off regularly;
• absolute channels 0 to 27 (light curves only), or approximately 1-10 keV for epoch 3 data; see Channel Energy Conversion Table.

• the Earth elevation angle is greater than 10 degrees (bright Earth effects have been seen near 5 degrees, thus the conservative cut);
• the pointing offset is less than 0.02 degrees;
• the time since the peak of the last SAA passage is greater than 30 minutes;
• the electron contamination is less than 0.1.

The results are named with the layer(s) extracted, the channel range (for lightcurves), and for background or net where appropriate. If a root name is specified, this will be prepended onto the default names. For example, Rex run with all defaults on the proposal P30401 will produce the following in the working directory:

combined products:

layer1.pha - layer 1 spectrum from all ObsIds
layer1_bkg.pha - layer 1 background spectrum from all ObsIds
layer1_ch0-27.lc - layer 1, ~1-10 keV light curve from all ObsIds
layer1_ch0-27_bkg.lc - layer 1, ~1-10 keV background light curve from all ObsIds
layer1_ch0-27_net.lc - layer 1, ~1-10 keV background subtracted light curve from all ObsIds

30401-01-01-00/ all of the above for one ObsId
30401-01-02-00/ all of the above for one ObsId
30401-01-03-00/ all of the above for one ObsId
30401-01-04-00/ all of the above for one ObsId
30401-01-05-00/ all of the above for one ObsId

aux/30401-01-01-00/FS4a_...._bkg - all background data files for this ObsId
aux/30401-01-01-00/basic.gti - list of Good Time Intervals for this ObsId
aux/30401-01-01-00/bkg.xdf - time ordered list of background files for this ObsId
aux/30401-01-01-00/std2.xdf - time ordered list of Standard 2 mode data files for this ObsId
for each ObsId, etc.

For HEXTE operation, the combined products will have the generic names hxt0_ch15-60.lc etc (for whatever channel range is chosen; the default range is the same as for the PCA: 0-27).

Basic Operation:

• To run from the command line with all defaults, give the path to the proposal directory and the -q flag to quell the prompting:

         %  rex -d /local/data/P30408 -q
  

  Rex will then run on the PCA data from every ObsId using all of the defaults specified above.

• To run from the command line changing the defaults, specify each parameter followed by the desired value:

         %  rex -d /local/data/P30408 -c 5-45 -l all -q
  

  This will tell Rex to extract all layers and to create full spectra and light curves from channels 5-45. The last flag, -q, tells it not to prompt for target, printmode, or root name, but rather take the defaults of "all", "both", and "none", respectively.

• To be prompted for all parameters except for the flags:

  
    %  rex
    Path to proposal level directory containing FMI: /local/data/P30408
    Target [all]:
    Printmode, l (ligh curve), s (spectra) [both]: s
    Channels for light curve extraction [0-27]:
    Layers to be extracted (1,2,3, or all) [1]: all
    Root name for products []: cenA
  

  Rex will then extract spectra from all layers for all targets in the proposal directory. It will prepend the string "cenA" onto all the output light curves and spectra. In this example, rex will produce in each resulting obsid directory the files cenA_layerall.pha and cenA_layerall_bkg.pha and the same at the top level for all obsids combined.

• To bypass time consuming steps which have already been run, Rex has two hidden flags, -f and -b, which respectively cause it to skip the xtefilt and pcabackest steps.

  Use -f only if you are sure that your filter files contain all columns needed by pcabackest (PCUn_ON, BKGD_THETA, BKGD_PHI) and maketime (all those listed in the selection expression in aux/expression.txt such as ELV, OFFSET, PCUn_ON, TIME_SINCE_SAA, and ELECTRON2.) If you have previously run xtefilt v1.6 or later (using the -s option to place the result in the standard products directory) with an up-to-date appidlist, you can safely use this flag to omit this step.

  Use -b only if Rex has already been run successfully on all the observations you are interested in. It will put the correct file lists where it needs them. If, however, Rex had a problem with one or more background files, it will not include them in the file lists. You can add them by hand if you've fixed the problem (see below) and re-run pcabackest yourself, or run Rex again, whichever is easier in the particular case.

• To analyze HEXTE data, Rex has the hidden flag -x. This tells Rex to analyze the HEXTE data for whichever cluster you supply as the argument to the -x flag. For example, if you have already run Rex once to create the filter files and perform a run on PCA Standard 2 data, your next step might be to do a HEXTE run:

         %  rex -f -d /local/data/P30408 -x 0 -c 15-60
  

  This will analyze HEXTE Cluster 0 data between channels 15 and 60. (Information not given on the command line will be prompted for.) It will skip the generation of the filter files (as these are already made), separate out background data using HXTBACK, extract spectra and perform the deadtime corrections using HXTDEAD, and merge the results. The default is to use the data from all four HEXTE detectors in the cluster chosen; note that one of the detectors in Cluster 1 no longer provides spectral information, so you may want to explicitly change the default when analyzing Cluster 1 data (see below).

Advanced usage:

• To change the selection expression used by maketime, you can edit the expression contained in the auxiliary file aux/expression.txt. This expression is passed directly to maketime and can take either C or Fortran syntax. Read the fhelp for maketime for more information about this tool.

  This is how one would change the requirement for certain PCUs to be on, e.g. to add a check for PCUs 3 and 4 if you have determined that they are on for your observations and you wish to add them to the extraction, or alternatively e.g. remove PCU 1 from the selection expression if you've determined that this PCU was turned off during your observation It is also where you can adjust the ELV and TIME_SINCE_SAA to be more or less conservative, as you see fit based on examining the data. If you want to examine slew data or occults, you would change the ELV and/or OFFSET criteria appropriately.

• To change the number of PCUs extracted, you can add or remove columns from the lists in the auxiliary files. For instance, to add PCUs 3 and 4 to the layer 1 extraction, add the columns

  X1LSpecPcu3
  X1RSpecPcu3
  X1LSpecPcu4
  X1RSpecPcu4
  

  to the text file aux/layer1.cols. Make sure to add a test to your maketime expression that these PCUs are on (as described above) to ensure a consistent set of data is extracted.

• To change the background model used, you can put the desired model files in the aux directory (or not), and edit the aux/model.files list to point to the new files, wherever they are. If you want to compare models, you will have to set up a separate results directory from which to run Rex (with a separate aux, of course.) Otherwise, Rex will overwrite previous background products, since all background files are simply named with "_bkg".

• To change the HEXTE detectors used, edit the aux/hexte.cols list. For example, to exclude the HEXTE detector in Cluster 1 that no longer provides spectral information, you would delete one line of this file to leave:

  SpecDet0
  SpecDet1
  SpecDet3
  

Post Rex:

Once you've used Rex to create source and background spectra, source and background light curves, etc, you will probably want to do a few more things. Before undertaking spectral analysis of PCA data, you'll need to create a response matrix for your spectrum using the pcarsp tool.

If you're new to X-ray data analysis, you'll want to study the online information about the XSPEC and XRONOS packages, for spectral and timing analysis respectively.

For further information about RXTE data and the FTOOLS suite available for their analysis, please see the extensive user guides on our data analysis pages, including our Getting Started Guide, the complete guide to RXTE data in The ABC of XTE, and especially the recipes in the RXTE Cook Book.

Rex attempts to trap most common errors and will print a warning to the screen if something generates an error. The warning will include the command line and tell where to find the log file for a complete record of the tool's output. Try cutting and pasting the command which caused the error and run it from the command line prompt. Use common sense and read the fhelp for the tool in question to diagnose the problem. A few specific examples are discussed below, but considering the number of different tools this script calls, the possibilities for error are numerous. If the error message is opaque, send the log of the particular problem along with a screen dump showing your inputs to xtehelp@athena.gsfc.nasa.gov.

• Not all observations in the proposal directory were extracted:

  Rex reads the index files to find the data; if these index files are not accurate, Rex may fail to find everything it should. The observations should be listed in the proposal level FMI, which can be updated using the tool recofmi if it is missing ObsIds that are in the proposal directory.

  Rex will check the good time intervals determined by maketime on the basis of the above criteria, and if there are no times matching these criteria for a given obsid, it will not attempt to extract any products. It will print a message to the screen to this effect when it hits such an obsid. Check the filtering criteria against the data by plotting from the filter file (the obsid/stdprod/FP*xfl file) the quantities found in aux/expression.txt. These criteria can be adjusted by editing that expression.

• How to recover from the general case of an error which causes an obsid or a single file to be skipped:

  One disadvantage of having a master script to go through a lot of computation autonomously is that a small problem in one section isn't easy to fill in after the script is finished. To make diagnosing problems easier, the log file that Rex writes when there is an error stores the entire command line, and this can be used to run by hand the particular job that failed. For instance, if an individual pcabackest run needs a larger timeslop than the default 128 seconds, the user can grab the original command from the error log, change the timeslop to be large enough, and run it again. But as with any pcabackest error, that background file won't have been written to the file list for that obsid or to the master list of all background files. The user can then edit the aux/obsid/bkg.xdf file and insert the missing entry with the correct path and in correct chronological order. Then, Rex can be run with the -b option to skip the background generation, and the extraction will then include the previously missing file.

  Note that if there are any problems with the background generations, they must be fixed before Rex can be run again with the -b option. The first time it is run, Rex determines whether pcabackest completed successfully before writing it to the lists to be extracted. If it is then run with the -b option, the individual obsid bkg.xdf lists remain the same with those problem files excluded, but the allbkg.xdf list is remade with the assumption that all Std2 files have corresponding, correct background files. (This is a necessary result of the ability of the script to be re-run with a different target extracted.) So any problems with those background files will be avoided in the individual obsid extractions, but will cause a problem with the combined extraction unless they are fixed by hand.

This page is maintained by the RXTE GOF and was last modified on Wednesday, 24-Aug-2022 11:10:30 EDT.