NAME
niobsmerge - merge NICER observation products
USAGE
niobsmerge infiles [outdir] ...
DESCRIPTION
The niobsmerge task merges observation products. This can be useful
when combining the products of several observation segments of the
same target that may span multiple days.
Please note that this task merges the output products of nicerl2, namely
the "ufa" file, the cleaned event file, and the filter file.
Also, optionally, the task will attempt to merge orbit files. It does
not merge all the original Level 1 data files of an observation directory.
The input is one or more NICER observation product output directories.
This is often a list of NICER observation directories, but it can also
use whatever "cldir" was used to invoke nicerl2.
The list of input directories is given by the "indirs" parameter.
Alternatively if indirs=NONE, then the task will expect an explicit
list of file names for each file type, as described below.
The output directory is given by the "outdir" parameter, which defaults
to the tasks current working directory.
Which files are used as inputs are governed by the inclfiles ("cl"
files), inufafiles ("ufa" files), inmkfiles ("mkf" or filter file),
and inorbfiles (orbit files) parameters. If indirs=NONE, then these
should be a comma-separated list of files to merge, otherwise, then
can be a pattern which describes how to find the files within each
input directory.
The default pattern are designed to match most
commonly used NICER processing outputs of nicerl2.
If no files of a given input type are found, that type is skipped.
The task can use some shorthand templates which allow the task to
avoid many hardcoded path names. These templates of the form $NAME
and can be used in most input or output file names to indicate certain
patterns are present without specifying them explicitly.
The available patterns are:
- $INDIR - each input directory as specified by the indirs parameter
- $OUTDIR - the output directory name
- $OBSID - the observation ID to use for output files as given by the obsid parameter
For example, a pattern of inufafiles="$INDIR/xti/event_cl/ni*_0mpu7_ufa.evt" will look
in each input directory in turn, searching for files of the requisite subdirectory and
file name pattern.
The output observation ID is given by the "obsid" parameter, and this is used to name
the output files. You can specify a different obsid with this parameter, or simply
specify output file names explicitly. If any output file name is listed as "NONE"
then that file type is skipped.
For each file type, the task will merge the input files that match
that type using the appropriate tool, and sort them by time. For
event files, NICER FPM selection and GTI information is preserved, so
the files can be used downstream for product and response extraction.
When you run nicerl3-spect or nicerl3-lc, the "indir" parameter of
those tools should be set to niobsmerge's outdir. See below for an
example.
By default, the task will not attempt to merge orbit files. Set
doorbfiles=YES to request that file type be merged.
PARAMETERS
- indirs="NONE" [string]
- A comma-separated list of NICER product directories, and @filename.lis
which specifies a list of directories, one per line. These directories
should typically be where nicerl2 created its output, which is commonly
the observation directories themselves. If indirs=NONE, then niobsmerge
usees file lists from inclfiles, inufafiles, inmkfiles and inorbfiles.
- (outdir=".") [string]
- Output directory name where merged products are placed.
The directory is created if it does not yet exist.
- (inclfiles="{$INDIR,$INDIR/xti/event_cl}/ni*_0mpu7_cl.evt{,.gz})") [string]
- Pattern matching string to find "cl" cleaned event files within
each indir. Use of standard Unix glob-matching patterns is allowed.
Also, certain shorthand patterns are allowed as documented above.
If indirs=NONE, then no shorthand patterns are allowed, and the user
is expected to provide a comma-separated list of files for this file
type, or an @filename.lis file containing a list of files.
- (inmkfiles="{$INDIR,$INDIR/xti/event_cl}/ni*_0mpu7_ufa.evt{,.gz})") [string]
- Pattern matching string to find "ufa" unfiltered event files within
each indir. Use of standard Unix glob-matching patterns is allowed.
Also, certain shorthand patterns are allowed as documented above.
If indirs=NONE, then no shorthand patterns are allowed, and the user
is expected to provide a comma-separated list of files for this file
type, or an @filename.lis file containing a list of files.
- (inorbfiles="{$INDIR,$INDIR/auxil}/ni*.mkf{,.gz})") [string]
- Pattern matching string to find filter ("mkf") files within
each indir. Use of standard Unix glob-matching patterns is allowed.
Also, certain shorthand patterns are allowed as documented above.
If indirs=NONE, then no shorthand patterns are allowed, and the user
is expected to provide a comma-separated list of files for this file
type, or an @filename.lis file containing a list of files.
- (inumkfiles="{$INDIR,$INDIR/auxil}/ni*.orb{,.gz})") [string]
- Pattern matching string to find orbit files within each indir.
Use of standard Unix glob-matching patterns is allowed. Also, certain
shorthand patterns are allowed as documented above. If indirs=NONE,
then no shorthand patterns are allowed, and the user is expected to
provide a comma-separated list of files for this file type, or an
@filename.lis file containing a list of files. Note that you must
also set doorbfiles=YES to cause orbit files to be merged.
- (obsid="merged") [string]
- Observation ID name that will be used in $OBSID pattern matching
in output file names. Note that the actual file OBSID_ID keywords are
not changed. Note that white space and punctuation characters are not
permitted: white space and periods are transformed to underscores, and
other punctuation is removed. To facilitate automated usage, specify an
obsid only with letters, digits and underscores.
- (clfile="$OUTDIR/ni$OBSID_0mpu7_cl.evt") [string]
- Name of output "cl" cleaned event file. A value of "NONE"
disables this output. Also, certain shorthand patterns are allowed as
documented above.
- (ufafile="$OUTDIR/ni$OBSID_0mpu7_ufa.evt") [string]
- Name of output "ufa" unfiltered event file. A value of "NONE"
disables this output. Also, certain shorthand patterns are allowed as
documented above.
- (mkfile="$OUTDIR/ni$OBSID.mkf") [string]
- Name of output filter ("mkf") file. A value of "NONE"
disables this output. Also, certain shorthand patterns are allowed as
documented above.
- (orbfile="$OUTDIR/ni$OBSID.orb") [string]
- Name of output orbit file. A value of "NONE" disables this
output. Also, certain shorthand patterns are allowed as documented
above. Please note that you must also set doorbfiles=YES to enable
this output.
- (doorbfiles=NO) [boolean]
- Set to "YES" to enable orbit file merging. Please note that orbit
files must be available in the input directories. If they are not
available, this output is skipped.
- (cleanup="YES") [boolean]
- If yes, then clean up temporary files. If no, temporary files
remain. This is typically for debugging.
- (clobber = NO) [boolean]
-
If the output file already exists, then setting "clobber = yes" will cause it to be overwritten.
- (chatter = 1) [integer, 0 - 5]
-
Controls the amount of informative text written to standard output.
Setting chatter = 4 or higher will produce detailed diagnostic output;
chatter = 1 prints out a basic diagnostic message. The default is to
produce a brief summary on output.
- (history = YES) [boolean]
-
If history = YES, then a set of HISTORY keywords will be written to
the header of the specified HDU in the output file to record the value
of all the niobsmerge task parameters that were used to produce the output
file.
EXAMPLES
Merge a set of observation directories data/[0-9]* and extract
spectral products. The outputs are placed in the directory
"outmerged", and standard file naming is assumed.
ls -d data/[0-9]* > indirs.lis
niobsmerge @indirs.lis outdir=outmerged obsid=merged clobber=YES
nicerl3-spect outmerged obsid=merged clobber=YES
Merge a set of files listed explicitly. Output files will have
standard naming.
ls -d data/[0-9]*/xti/event_cl/ni*_0mpu7_cl.evt* > clfiles.lis
ls -d data/[0-9]*/xti/event_cl/ni*_0mpu7_ufa.evt* > ufafiles.lis
ls -d data/[0-9]*/auxil/ni*.mkf* > mkfiles.lis
niobsmerge indirs=NONE outdir=outmerged inclfiles=@clfiles.lis \
ufafiles=@ufafiles.lis mkfiles=@mkfiles.lis clobber=YES
Merge orbit files in addition to the other file types.
ls -d data/[0-9]* > indirs.lis
niobsmerge @indirs.lis outdir=outmerged clobber=YES doorbfiles=YES
Disable output of UFA files.
ls -d data/[0-9]* > indirs.lis
niobsmerge @indirs.lis outdir=outmerged clobber=YES ufafile=NONE
Specify explicit names for output files, and disable UFA output.
ls -d data/[0-9]* > indirs.lis
niobsmerge @indirs.lis clfile=cleaned.evt ufafile=NONE mkfile=merged.mkf clobber=YES
SEE ALSO
nimkfmerge
niextract-events
nicerl2
nicerl3-spect
nicerl3-lc
LAST MODIFIED
May 2023