NAME

sgdpipeline - Hitomi SGD reprocessing task

USAGE

sgdpipeline indir outdir steminputs entry_stage exit_stage

DESCRIPTION

The sgdpipeline task duplicates most of the Hitomi pipeline for SGD (in particular, not the trend data). It allows the user to run all or part of the pipeline processing and to vary the calibration files and filtering (screening) criteria used. A number of other pipeline processing parameters can also be changed.

SGD Pipeline Stages

The sgdpipeline is divided into 2 stages:

  1. Calibration
  2. Data screening
Each stage may be run singly or in combination with the preceding stages. This is controlled by the 'entry_stage' and 'exit_stage' parameters.

SGD Stage 1 consists of the following steps:

  1. Run hxisgdsff
  2. Run hxisgdpha
  3. Run sgdevtid

The data screening (Stage 2) is identical to that in the production pipeline, when default parameters are used. For details on the default screening applied to the SGD events, see:

Default GTI used for screening data are:

The cleaning process occurs twice for SGD: once to produce cleaned event files, and again to produce pseudo event files.

INPUT

The input to sgdpipeline is specified using (at minimum) the 'indir' parameter. This should be specified as the SGD event_uf level sequence directory, e.g.:

sgdpipeline indir=/path/to/100039010/sgd/event_uf

Paths to specific SGD housekeeping and satellite data files can be specified using the 'attitude', 'housekeeping', 'extended_housekeeping' 'makefilter', 'orbit', and 'obsgti' parameters. The attitude, orbit, and SGD housekeeping files are required for Stage 1 calibration.

OUTPUT

File names, etc.

The number of output files depends on both pipeline processing stage(s) and the options selected. All output files are written to the directory specified by the 'outdir' parameter. The archive directory structure is NOT reproduced (i.e. all output files are in a single directory).

The names of files produced are the same as those found in the HEASARC archive. However, the usual ahXXXXXXXXX prefix, where "XXXXXXXXX" is the sequence number, can be replaced by a character string set by the 'stemoutputs' parameter. This defaults to the value set by the 'steminputs' parameter.

PARAMETERS

indir = indir/ [string]
Input directory.

outdir = outdir/ [string]
Output directory.

steminputs = ah1001 [string]
Stem inputs.

stemoutputs = ah1001 [string]
Stem outputs.

instrument = SGD [string SGD|SGD1|SGD2]
Instrument name.

entry_stage = 1 [integer 1|2]
Entry stage.

(exit_stage = 2) [integer 1|2]
Exit stage.

(sgd_start = 0.0) [real]
Earliest validity time for SGD calibration files.

verify_input = no [boolean yes|no]
If 'verify_input=yes', input FITS files are verified with ftverify.

(sgd_mkflabel = SGDSFFA1#) [string]
Label to use for SGD MKF GTI creation. The hash is replaced by one of the following strings corresponding to the appropriate Compton camera: CC1, CC2, or CC3. For pseudo events PSE is appended to the end of the label.

(sgd_ehklabel = SGDSFFA1#) [string]
Label to use for SGD event screening. The hash is replaced by one of the following strings corresponding to the appropriate Compton camera: CC1, CC2, or CC3. For pseudo events PSE is appended to the end of the label.

(sgd_evtlabel = SGDSFFA1#) [string]
Label to use for SGD event screening. The hash is replaced by a respective Compton camera: CC1, CC2, or CC3. For pseudo events "PSE" is appended to the end of the label.

(ra = -999.99999) [real]
Right ascension [degrees] of the point to appear in the center of SKY coordinate images. If 'ra' is within the range 0 <= 'ra' <= 360, the value of the parameter is used and is written to the keyword RA_NOM in output files. 'ra' is outside this range, the 'ra' value is read from the event header of the input file, by searching first for the RA_NOM keyword and if found using its value, or if not found, by searching for the RA_PNT keyword and using its value. If neither keyword is found, the called task coordevt exits with an error.

(dec = -999.99999) [real]
Declination [degrees] of the point to appear in the center of SKY coordinate images. If 'dec' is within the range -90 <= 'dec' <= +90, the value of the parameter is used and is written to the keyword DEC_NOM in output files. If 'dec' is outside this range, the 'dec' value is read from the event header of the input file, by searching first for the DEC_NOM keyword and if found using its value, or if not found, by searching for the DEC_PNT keyword and using its value. If neither keyword is found, the called task coordevt exits with an error.

(roll = 0.0) [real]
Roll angle [degrees] about the center of the SKY coordinate system. The roll angle is the angle measured counterclockwise from Celestial North to the positive SKY Y axis.

(sg1_optfocx = -999.99999) [real]
SGD1 optical axis FOCX coordinate.

(sg1_optfocy = -999.99999) [real]
SGD1 optical axis FOCY coordinate.

(sg1_optskyx = -999.99999) [real]
SGD1 optical axis DETX coordinate.

(sg1_optskyy = -999.99999) [real]
SGD1 optical axis DETY coordinate.

(sg2_optfocx = -999.99999) [real]
SGD2 optical axis FOCX coordinate.

(sg2_optfocy = -999.99999) [real]
SGD2 optical axis FOCY coordinate.

(sg2_optskyx = -999.99999) [real]
SGD2 optical axis DETX coordinate.

(sg2_optskyy = -999.99999) [real]
SGD2 optical axis DETY coordinate.

(sg1_ra_pnt = -999.99999) [real]
RA [degrees] of SGD1 pointing.

(sg1_dec_pnt = -999.99999) [real]
DEC [degrees] of SGD1 pointing.

(sg2_ra_pnt = -999.99999) [real]
RA [degrees] of SGD2 pointing.

(sg2_dec_pnt = -999.99999) [real]
DEC [degrees] of SGD2 pointing.

(extended_housekeeping = ah1001.ehk) [string]
Parameter for ahgtigen: Extended housekeeping file.

(makefilter = ah1001.mkf) [string]
Parameter for ahgtigen: Makefilter file.

(obsgti = ah1001_gen.gti) [string]
Parameter for ahscreen: Observation GTI file.

(remapfile = CALDB) [filename CALDB|file name]
Parameter for hxisgdsff, sgdevtid: File describing the remapping of ASIC_ID and READOUT_ID to sequential numbers in the columns ASIC_ID_RMAP and READOUT_ID_RMAP. If the parameter is set to CALDB, the file is read from the calibration database.

(gainfile = CALDB) [filename CALDB|file name]
Parameter for hxisgdpha: FITS table describing the gain calibration of PHA for the HXI or SGD camera. The domain of PHA values is divided into intervals, and a set of cubic polynomial coefficients is given for each interval. If the parameter is set to CALDB, the file is read from the calibration database.

(badpixfile = CALDB) [filename CALDB|file name]
Parameter for hxisgdpha, sgdevtid: File containing the energy threshold defining a real signal for each readout.

(fluorefile = CALDB) [filename CALDB|file name]
Parameter for sgdevtid: Calibration file containing the energy ranges of fluorescence electrons in CdTe and Si.

(probseqfile = CALDB) [filename CALDB|file name]
Parameter for sgdevtid: Calibration file assigning the probability of each possible sequence of interactions.

(probfovfile = CALDB) [filename CALDB|file name]
Parameter for sgdevtid: Calibration file containing the probability for photons to be in the SGD camera field of view, based on incident angle.

(leapsecfile = REFDATA) [filename CALDB|REFDATA|file name]
Name of the leap second file. If set to CALDB or REFDATA, the file in the calibration database or in the REFDATA area, respectively, is used.

(selectfile = CALDB) [filename CALDB|file name]
The file containing selection expressions. If 'selectfile' is set to CALDB, the file is read from the calibration database.

(outnsubcol = no) [boolean yes|no]
Parameter for hxisgdpha: Output the PHA_NSUB column. This column is the PHA minus the ASIC common-mode noise (ASIC_CMN column in the SFF).

(datamode = NONE) [string]
Parameter for hxisgdpha: If 'datamode' is not set to NONE (or an empty string), 'datamode' is used in place of the value of the keyword DATAMODE that is in the input event file.

(rejectbgo = no) [boolean yes|no]
Parameter for sgdevtid: If 'rejectbgo=yes', reject events for which BGO trigger occurred. The BGO is the active anticoincidence shield made of bismuth germanium oxide.

(skipreco = no) [boolean yes|no]
Parameter for sgdevtid: If 'skipreco=yes', READALL and CALMODE occurrences are not reconstructed.

(outtracefile = NONE) [filename NONE|file name]
Parameter for sgdevtid: Output reconstruction tracing file.

(numsignal = 48) [integer]
Parameter for sgdevtid: Maximum number of signals to analyze. Reconstruction is not attempted on occurrences with more than this number of signals.

(d10 = 3.2) [real]
Parameter for sgdevtid: Orthogonal distance [mm] between two adjacent pixels in the same detector layer; used in identifying charge sharing events.

(d1a1a = 5.0) [real]
Parameter for sgdevtid: Diagonal distance [mm] between two adjacent pixels in same detector layer; used in identifying possible CdTe-CdTe fluorescence signals in a single layer.

(d1a1b = 5.0) [real]
Parameter for sgdevtid: Distance [mm] between two CdTe layers; used in identifying possible CdTe-CdTe fluorescence signals in different layers.

(d1a2 = 14.0) [real]
Parameter for sgdevtid: Distance [mm] between CdTe and Si layers; used in identifying possible Si-CdTe fluorescence signals.

(d1a3 = 5.0) [real]
Parameter for sgdevtid: Distance [mm] between Si layers; used in identifying possible electron scattering signals.

(a = 3.0) [real]
Parameter for sgdevtid: Acceptance tolerance for F test in Step 2.

(b = 3.0) [real]
Parameter for sgdevtid: Acceptance tolerance for G test in Step 2.

(probaccept2 = 0.1) [real]
Parameter for sgdevtid: Probability threshold for M = 2 acceptance in Step 3.

(probaccept3 = 0.1) [real]
Parameter for sgdevtid: Probability threshold for M = 3 acceptance in Step 3.

(probaccept4 = 0.1) [real]
Parameter for sgdevtid: Probability threshold for M = 4 acceptance in Step 3.

(distz = 1000000.0) [real]
Parameter for sgdevtid: Very large distance [mm] of target object in Step 4. The Z-direction is toward the sky; this distance represents the distance to an object far away from the detector.

(paraoffset0 = 1.6) [real]
Parameter for sgdevtid: Offset value used in step 4 to tune G[k,0], which is a component of the figure of merit for multiple-hit sequences.

(paraoffset1 = 1.0) [real]
Parameter for sgdevtid: Offset value used in step 4 to tune G[k,1], which is a component of the figure of merit for multiple-hit sequences.

(paraoffset2 = 1.0) [real]
Parameter for sgdevtid: Offset value used in step 4 to tune G[k,2], which is a component of the figure of merit for multiple-hit sequences.

(weight0 = 1.0) [real]
Parameter for sgdevtid: Weight applied to G[k,0] in calculating the figure of merit for multiple-hit sequences.

(weight1 = 0.0) [real]
Parameter for sgdevtid: Weight applied to G[k,1] in calculating the figure of merit for multiple-hit sequences.

(weight2 = 0.0) [real]
Parameter for sgdevtid: Weight applied to G[k,2] in calculating the figure of merit for multiple-hit sequences.

(weight3 = 0.0) [real]
Parameter for sgdevtid: Weight applied to Prob[k], which is a component of the figure of merit for multiple-hit sequences.

(delgmethod = ANALYTIC) [string ANALYTIC|CORNER]
Parameter for sgdevtid: Method to be used to calculate the error in cos(theta_G) in the test of consistency between geometric and kinematic scattering angles.

(occurrenceid = -1) [integer]
Occurrence to process (if >0).

(randomize = yes) [boolean yes|no]
Parameter for hxisgdpha: Allow randomization of fractinal part of PHA before calibration.

(seed = 0) [integer]
Random number generator seed; uses system time for 'seed=0'.

(stemreport) [string]
File stem for log and temporary files. If the parameter is not set, the script is automatically set the stem to "sgdpipeline_YYYYMMDDTHHMMSS_" and appends log file and temp file names as necessary. Intended to be set by ahpipeline.

(numerrs = 0) [string]
Number of errors from sgdpipeline (output).

(cleanup = yes) [boolean yes|no]
Delete temporary files if 'cleanup=yes'.

(clobber = no) [boolean yes|no]
Overwrites the existing output file if set to yes.

(chatter = 2) [integer 0|1|2|3]
Chatter level for output. Set to 0 to suppress output, or to 1, 2, or 3 for increasing the chatter of the output.

(logfile = !DEFAULT) [string DEFAULT|NONE|file name]
Log file name. If set to DEFAULT, uses the name of the task and, if preceded by "!", overwrites the file if it exists. If set to NONE, no log file is created.

(debug = no) [boolean yes|no]
Diagnostic output is printed to the screen if set to yes.

(history = yes) [boolean yes|no]
Records task parameters in HISTORY.

THE FOLLOWING PARAMETERS ARE SET BY THE SCRIPT INTERNALLY:

HXISGDSFF:

infile [filename]
Set based on the unfiltered event files found in the 'indir' directory during initialization. Input First FITS File (FFF) from HXI or SGD.

outfile [filename]
Set to the same name as 'infile', but in the 'outdir' directory. Output Second FITS File (SFF). May be the same or a different file from infile.

HXISGDPHA:

infile [filename]
Set based on the unfiltered event files found in the 'indir' directory during initialization and processed through hxisgdsff. Input SFF from HXI or SGD.

outfile [filename]
Set to the same name as 'infile', but in the 'outdir' directory. Output SFF.

HXISGDEXPAND:

infile [filename]
Set based on the unfiltered event files found in the 'indir' directory during initialization and processed through hxisgdsff and hxisgdpha. Input file of event data from an HXI instrument. This is the SFF after PHA calibration by the task hxisgdpha.

outfile [filename]
Set to the same name as 'infile', but in the 'outdir' directory and changing the extension from "_uf.evt" to "exp_ufa.evt". Output event file containing results of the event reconstruction.

SGDEVTID:

infile [filename]
Set based on the unfiltered event files found in 'indir' directory during initialization and processed through hxisgdsff and hxisgdpha. Input file of event data from an HXI instrument. This is the SFF after PHA calibration by the task hxisgdpha.

outfile [filename]
Set to the same name as 'infile', but in the 'outdir' directory and changing the extension from "_uf.evt" to "rec_ufa.evt". Output event file containing results of event reconstruction.

AHGTIGEN:

outfile [filename]
Set to the concatenation of 'sgdfile_out' and the extension ".gtimkf" when the input file is set to the makefilter file parameter. Set to the concatenation of 'sgdfile_out' and the extension ".gtiehk" when the input file is set to the extended_housekeeping file parameter.

(instrume = NONE) [string SGD1|SGD2|NONE]
Read from the header of the file being processed.

(label = NONE) [string]
The concatenation of "SGDSFFA1." and 'detnam' or of "SGDSFFA1.", 'detnam', and ".PSE", depending on if the files being generated are "clean" or "pseudo", respectively.

AHSCREEN:

infile [string]
Set systematically to the elements in an array which is constructed from any reconstructed unfiltered event files produced in the calibration stage. If the calibration stage is being skipped, this array is filled from searching the input directory instead.

outfile [filename]
Set to the same name as 'infile', but in the 'outdir' directory and changing the extension from "rec_ufa.evt" to "rec_cl.evt".

(gtifile) [string]
Set as an array of gtifiles with corresponding extensions: POINTING, TEL, MKF, and (where applicable) EHK.

(upkeyword = yes) [boolean yes|no]
Update timing keywords from input file(s) in output file (hardcoded as yes).

(label = NONE) [string]
The concatenation of "SGDSFFA1." and 'detnam' or of "SGDSFFA1.", 'detnam', and ".PSE", depending on if the files being generated are "clean" or "pseudo", respectively.

EXAMPLES

  1. The following command re-calibrates (Stage 1) and re-screens (Stage 2) all SGD data for sequence 100039010 that currently resides in the directory /data/100039010/sgd/event_uf, and the output are stored in a directory called /data/100039010_reproc/. 
       sgdpipeline indir=/data/100039010/sgd/event_uf outdir=/data/100039010_reproc entry_stage=1 exit_stage=2 \
      steminputs=ah100039010 obsgti=/data/100039010/auxil/ah100039010/ah100039010_gen.gti \
      makefilter=/data/100039010/auxil/ah100039010/ah100039010.mkf \
      exended_housekeeping=/data/100039010/auxil/ah100039010/ah100039010.ehk
  2. The following command re-screens (Stage 2 only) SGD data for the same data set as in the previous example. 
    sgdpipeline indir=/data/100039010/sgd/event_uf outdir=/data/100039010_reproc entry_stage=2 exit_stage=2 \
      steminputs=ah100039010 obsgti=/data/100039010/auxil/ah100039010/ah100039010_gen.gti \
      makefilter=/data/100039010/auxil/ah100039010/ah100039010.mkf \
      exended_housekeeping=/data/100039010/auxil/ah100039010/ah100039010.ehk

SEE ALSO

ahcalctime, ahpipeline, hxisgdsff, hxisgdpha, sgdevtid, ahfilter, ahgtigen, ahscreen

LAST MODIFIED

November 3, 2023