NAME
nimaketime - Create standard NICER screening GTI
USAGE
nimaketime infile outfile ...
DESCRIPTION
The nimaketime task creates a GTI file for event screening based
on standard NICER screening criteria.
nimaketime does not actually screen events; rather it creates a
GTI file that can be used to screen events using a task like
extractor
or niextract-events
nimaketime is a helper script that can construct a filtering
expression for you. You can also run maketime yourself with
the same expressions.
nimaketime allows you to input one or more GTIs to be combined
with "AND" (intersection) merging. This allows you to bring
in previous screening to be combined with the nimaketime screening.
You can also give a custom filtering expression to be applied
to the .mkf file which is not provided by nimaketime.
SCREENING CRITERIA
nimaketime can apply the following screening criteria. The exact filtering string
is also given in quotes next to each item.
SAA
The following options are used for screening out times when NICER
passes through the South Atlantic Anomaly (SAA). The SAA is a source
of very high background and may also distort the NICER energy scale
calibration.
Use either nicersaafilt=YES or saafilt=YES, but not both.
- nicersaafilt=YES. Excludes times within SAA as defined by NICER_SAA. This is a NICER-specific definition of the SAA stored in CALDB.
"(NICER_SAA==0)
The NICER_SAA contour is smaller than the standard SAA contour, but has been customized for NICER's environment.
- saafilt=NO. When YES, excludes times within the generic SAA defined by prefilter.
"(SAA==0)
The SAA contour is larger (more conservative) than NICER_SAA, and may be useful for faint targets where conservative screening is desired.
This is "no" by default because normally you do not need to apply both the generic and NICER-specific screening.
Pointing and On-Target Tracking
These options select times when NICER is on-target and tracking a target above the horizon.
- trackfilt=YES. Excludes times when NICER is not in target tracking mode.
(ATT_MODE==1 && ATT_SUBMODE_AZ==2 && ATT_SUBMODE_EL==2)
This is on by default to select data when NICER is tracking a target.
- ang_dist=DIST. Excludes times when the angular distance between the target and NICER's boresight is greater than DIST.
(ANG_DIST < DIST)
The pointing constraint is currently DIST=0.015 (within 0.9 arcmin of target). This may need to be loosened on rare occassions when pointing is compromised.
See "DEALING WITH OFFSET POINTING" below for more information about offset pointing.
- st_valid=YES. Excludes time when star tracker solution is not valid.
(ST_VALID == 1)
This option was added in nimaketime version 1.4, and will exclude times when the NICER tracking system estimated that it was on-target
but the star tracker did not provide valid solution. The NICER team generally recommends the screening cut.
- elv=MINELV. Excludes times when the distance to the earth limb is less than MINELV.
(ELV > MINELV)
The default elevation constraint is MINELV=15 degrees. This was changed from 30 degrees in version 1.3 and earlier. The primary reason
to use this constraint is to avoid absorption and fluorescence by Earth's atmosphere and optical loading by reflected light.
- br_earth=MIN_BR_EARTH. Exclude times when distance to the bright earth is less than MIN_BR_EARTH.
(BR_EARTH > MIN_BR_EARTH)
The default bright earth constraint is MIN_BR_EARTH=30 degrees. This was changed to 30 from 40 degrees in version 1.3 and earlier.
The primary reason to use this constraint is to avoid optical loading by reflected light.
Background and Detector Performance
These options select for reduced background and optical loading and
ensure that at least some detectors are enabled.
- cor_range=A-B. Specify a range of magnetic cut-off rigidity (COR) to accept (in GeV/c). This can be used for filtering of high background regions. You can use the notation cor_range="A-" to specify a minimum but no maximum, cor_range="-B" to specify a maximum but no minimum, and the default cor_range="-" accepts all CORs.
(COR_SAX > A && COR_SAX < B)
NICER background is generally correlated with the geomagnetic location of the observatory, and lower numbers indicate higher background. By default
the NICER team does not have a recommended COR range, but instead uses overshoot cuts (see below).
- min_fpm=MIN_FPM. Specify a minimum number of enabled detectors.
(NUM_FPM_ON > MIN_FPM)
The default value for this is MIN_FPM=7. This was changed from 38 in versions 1.3 and earlier. The reason for this
cut is to ensure that at least some detectors are enabled, and exclude times when the full array was off.
- underonly_range=A-B. Excludes times when undershoot count rate is out of range A-B counts per second per FPM.
(FPM_UNDERONLY_COUNT > A && FPM_UNDERONLY_COUNT < B)
NICER's performance, especially the energy gain scale, is affected by optical loading. Reduce optical loading effects by
reducing the default range 0-500 to a smaller range (0-50 is optimal). This option was added in version 1.4
- overonly_range=A-B. Excludes times when overshoot count rate is out of range A-B counts per second per FPM.
(FPM_OVERONLY_COUNT > A && FPM_OVERONLY_COUNT < B)
NICER's background level is correlated with "overshoots", which are saturating particle detections. The default
range of 0-30 will exclude most strong background flare-ups. Note that this cut is applied in addition to overonly_expr.
This option was added in version 1.4.
- overonly_expr=EXPR. Excludes times when FPM_OVERONLY_COUNT exceeds EXPR. EXPR is any maketime expression using
columns from the filter file.
(FPM_OVERONLY_COUNT < EXPR)
This applies additional screening based on environment-specific conditions, using columns from the filter file. The
default value is currently "NONE" indicating to not use a special expression. In versions of nimaketime from
versions 1.4 through 1.10, the expression was ,
overonly_expr="1.52*OVERONLY_NORM*COR_SAX**(-0.633)" (OBSOLETE)
which was heuristically determined by the NICER team to exclude most background flares in different geomagnetic conditions. Because background models are now included in NICERDAS that can handle modest background increases, this
expression has been changed to "NONE".
This option was added in version 1.4.
Note that the name OVERONLY_NORM is taken from the maximum value of overonly_range (or 1.0 if
no maximum was given).
- max_lowmem=m. Excludes times when TOT_LOWMEM_FIFO exceeds
the value specified.
(TOT_LOWMEM_FIFO < m)
This column indicates when the MPU is
discarding events due extremely high count rates, in which case the
calibration problems will likely arise.
The default value of 0 disables this criterium, in favor of a similar
screening done in niautoscreen.
- threshfilter=(ALL|NIGHT|DAY). A shortcut to selected desired threshold range,
as described below. Use either threshfilter or thresh_range, but not both.
threshfilter=ALL corresponds to thresh_range=*-*
threshfilter=NIGHT corresponds to thresh_range=-3-3
threshfilter=DAY corresponds to thresh_range=32-38
The cleanest data will be found with threshfilter=NIGHT. Note that before the
optical light leak in May 2023, the threshold was always at its "night" setting,
so threshfilter=NIGHT will select all data, regardless of whether it was observed
in orbit day or orbit night.
- thresh_range=A-B. Only includes times when the relative NICER threshold is within
the given range.
(DELTA_SLOW_LLD > A && DELTA_SLOW_LLD < B)
The value reported in the DELTA_SLOW_LLD column gives the relative threshold
setting. Here relative means relative to the nominal post-launch threshold,
and threshold setting means the read-back value of the threshold, in units
of the commanded threshold setting. Typically this value is ~0 within tight
tolerances, and the default range is -3 to +3 (which is approximately within
12 eV of nominal). Set the range to -3-38 to accomodate data taken when
NICER experienced a large light leak in 2023. However, mixing thresholds
must be done with extreme caution.
GTI Filtering
These options filter GTI entries as described below.
- min_gti=m. Specify a minimum duration for a GTI of m seconds.
- erodedilate=e. Specify an erodedilate parameter of e seconds.
DEALING WITH OFFSET POINTING
Occasionally, NICER performs offset pointing to avoid a contaminating
spoiler target. This may also occassionally occur if the incorrect
target coordinates were used and/or a preliminary target position was
used for initial observations.
In that case, the standard ang_dist=NNNN filtering criterium may not
return any valid data. This is because the offset pointing location
is more than the specified amount (default 0.015 deg = 54 arcsec).
If the goal is to get some data, regardless of whether the data
is high quality or on-axis, then using the following filtering expression:
nimaketime ... ang_dist=180 expr="ANGSEP(RA,DEC,#RA_NOM,#DEC_NOM)<0.015"
This expression will accept pointing within 54 arcseconds of the nominal
pointing direction rather than the nominal target direction. The value
of 0.015 can be increased to a larger amount if necessary.
ADDITIONAL GEOGRAPHIC SCREENING
Users may wish to apply additional screening criteria based on
geographic position. Since this can be difficult, nimaketime provides
some shorthand capability for geographic screening.
The screening should be the form of a region file, and on the command
line as latlonregfile=filename.reg. The region file should in the
format of a standard spatial region file, and specify the contour of
good or bad data. The region file should specify longitude in the
range SAT_LON=-180 to SAT_LON=+180; this is done to allow easy
filtering of the South Atlantic Anomaly (SAA), which crosses
SAT_LON==0.
Note that this screening will be applied in addition to the other
screening criteria listed above. If you want to avoid
double-screening of SAA, you may want to set nicer_saa=NO and saa=NO.
Users can also use standard regions that are located in NICER CALDB.
Currently there are two named regions in CALDB:
- SAACONT - standard NICER contour
- SAACONTLARGE1 - enlarged NICER contour
Specify a CALDB contour using latlonregfile=CALDB:SAACONT or
latlonregfile=CALDB:SAACONTLARGE1.
If the sense of the region file should be inverted to keep good data,
then use latlonreginvert=YES. This is true for SAA filtering.
For example, for enlarged SAA region filtering, use the following options:
nimaketime ... latlonregfile=CALDB:SAACONTLARGE1 latlonreginvert=YES nicer_saa=NO
DIAGNOSING LOST DATA
nimaketime can help diagnose the reason for lost data. It
can print the amount of good time after certain stages
of filtering.
It does this if calc_intermediate=YES and chatter is 2 or more.
It does not interfere with the final calculation of good time.
If this process ends up using too much computing time, for example
if your filter file is very large, you can disable the
intermediate calculation by setting setting chatter to 0 or 1,
or by setting calc_intermediate=NO.
If enabled, the task prints a table which shows the remaining good
time after a set of filtering stages. Each good time printed is
cumulative, including all the previous stages. The values shown
include the remaining good time, as well as the number of
good time intervals.
The stages that can be printed include:
- On-Target + Observing - pointing control is trakcing on target
(trackfilt), target is in the field of view (given by ang_dist),
target is above the earth and br_earth elevations, star tracker
solution is valid (st_valid) and detectors are enabled
(min_fpm).
- LOWMEM + Threshold Quality - total number of MPU lost events
is less than min_lowmem, and FPM relative threshold setting is
within range given by thresh_range. Note that older filter
files may not have the required columns, and in that case these
filtering steps are skipped.
- SAA + COR + Orbit - object is not in SAA (nicersaa/saa), and
not in keep-out region if provided by latlonregfile, and COR_SAX
is within cor_range.
- Undershoots - MPU_UNDERONLY_COUNTS are within the range given
by underonly_range.
- Overshoots - MPU_OVERONLY_COUNTS are within the range given
by overonly_range.
- User Expression - any filtering by the user screening
expression (expr).
- User GTI - any additional reduction after applying the
user-specified good time intervals (ingtis)
- Final Merged Screening - the final resulting good time.
DEALING WITH MANY TINY GTIs
In some cases a set of filtering criteria results in many, tiny
GTIs. This often occurs if the filtering criteria sample a noisy
distribution such that the filtered value is sometimes inside the
range and sometimes outside of the range. The result may be
many GTIs that are 1-2 seconds.
As of nimaketime version 1.12, the task has a way to combat this
problem by filtering the GTI rows. The task uses the capability of
ftadjustgti to remove small GTIs. It uses the capability called
"erodedilate" which is first reduces each GTI by a set amount and
then increases each GTI by the same amount. The net result is that
small GTIs are removed.
Note that this process will effectively remove any GTI smaller than
the erodedilate setting (in seconds) x 2. nimaketime acts to
conservatively remove (rather than heal) any "shredded" GTI
situations.
The process described above apply to any telemetry-based selections.
Any user-suppled expressions (expr) or input GTIs (ingtis) are used
as-is with no GTI filtering.
PARAMETERS
- infile [filename]
- Input prefilter (.mkf) file name.
- outfile [filename]
- Output GTI file name.
- (nicersaafilt = "YES") [boolean]
- Apply NICER_SAA filtering. See above.
- (saafilt = "NO") [boolean]
- Apply SAA filtering. See above.
- (trackfilt = "YES") [boolean]
- Apply pointing tracking filtering. See above.
- (onlytrackfilt = "NO") [boolean]
- If no, then only apply trackfilt=YES and disregard all other types of filtering.
This is intended only for internal pipeline bookkeeping and not recommended for
general scientist usage.
- (st_valid = "YES") [boolean]
- Require star tracker valid solution?
- (ang_dist=0.015) [real]
- Apply ANG_DIST filtering (degrees). ANG_DIST must be less than this value. The default of 0.015 degrees is 54 arcseconds. See above for more information.
- (elv=15) [real]
- Apply ELV filtering (degrees). ELV must be greater than this value. See above.
- (br_earth=30) [real]
- Apply BR_EARTH filtering (degrees). BR_EARTH must be greater than this value. See above.
- (cor_range="*-*") [string]
- Apply COR_SAX filtering (GeV/c). Specify a range A-B. See above.
- (underonly_range="0-500") [string]
- Apply undershoot count rate range per module. Specify a range A-B. See above.
- (overonly_range="0-30") [string]
- Apply overshoot count rate range per module. Specify a range A-B. See above.
- (overonly_expr="NONE") [string]
- Apply curve-base overshoot filtering as an expression, or NONE for no additional
overonly expression-based screening. See above.
- (min_fpm=7) [integer]
- Apply NUM_FPM_ON filtering. See above.
- (calc_intermediate=YES) [boolean]
- Calculate intermediate good time summary statistics? If yes,
then nimaketime calculates the Good Time Intervals after several
intermediate stages of screening. The final output is not
changed. Intermediate values are only printed if chatter is
greater than or equal to 2. Note that the total good time
values are successively cumulative, and include all the filtering
from that stage and previous stages.
Set calc_intermediate=NO if
nimaketime is using too many computing resources.
- (ingtis="NONE") [string]
- Apply additional GTI filters. If you have done external filtering and
produced a GTI file, you can merge with the maketime GTI. Either a comma-separated
list of files or an @filelist.lis file. All GTIs (including the maketime GTI) are combined
using "AND" (intersection) merge. A value of "NONE" indicates no additional
GTI filtering.
- (max_lowmem=0) [real]
- Maximum value of TOT_LOWMEM_FIFO lost events, in units of events
per second for the entire NICER array. A value of 0 disables this
screening criterium.
- (thresh_range="-3.0-3.0") [string]
- Range of NICER lower-level threshold settings to include.
This is the array-average relative threshold setting, where
0.0 indicates the nominal threshold level, and the units
correspond to the flight relative threshold offset value.
For NICER optical light leak conditions, it would be typical
to use a range of "-3.0-38" to include both nominal and "+35"
settings, and "32-38" to include just the daytime threshold
settings.
- (threshfilter="NIGHT") [string]
- A shortcut to select a certain desired threshold range,
as described above. One of "ALL", "DAY" or "NIGHT".
- (mingti=5.0) [real]
- Minimum GTI size in seconds for GTI filtering. This applies to
any telemetry-selected filtering criteria (i.e. not user expressions
or user GTIs).
- (erodedilate=5.0) [real]
- Specify the "erodedilate" parameter for GTI filtering of GTI
rows. This applies to any telemetry-selected filtering criteria
(i.e. not user expressions or user GTIs).
- (expr="NONE") [string]
- Apply additional maketime expressions, beyond the standard criteria.
This should be a standard maketime filtering expression that evaluates to
true when conditions are good. A value of "NONE" indicates to use only
the standard filtering.
- (latlonregfile="NONE") [string]
- Apply an additional geographic screening as described above. Use
"NONE" to skip this step, or "CALDB:name" to query CALDB for region named
"name".
- (latlonreginvert=NO) [boolean]
- If NO, then the region file encloses good data. If YES, then the region
file should enclose bad data.
- (outexprfile="NONE") [string]
-
nimaketime will write its filtering expressions to this file so that the user
can inspect them, or to use them in downstream processing. A value of
"NONE" indicates to not save the filtering expressions.
- (cleanup="YES") [boolean]
- If yes, then clean up temporary files. If no, temporary files
remain. This is typically for debugging.
- (clobber = NO) [boolean]
-
If the output file already exists, then setting "clobber = yes" will cause it to be overwritten.
- (chatter = 2) [integer, 0 - 5]
- Amount of verbosity of the task. For chatter=0, no output is
printed. For chatter=5, debugging output is
printed.
- (history = YES) [boolean]
- If history = YES, then a set of HISTORY keywords will be written to
the header of the specified HDU in the output file to record the value
of all the task parameters that were used to produce the output
file.
EXAMPLES
1. Apply standard NICER screening criteria.
nimaketime ni1706260815.mkf good.gti
This example will supply standard screening as detailed above.
2. Apply more custom screening
nimaketime ni1706260815.mkf good40.gti elv=40 ang_dist=0.005 ingtis=extra.gti
This example applies the standard screening criteria, with the exceptions that
the minimum ELV elevation is set to 40 degrees, the maximum angular distance
is set to 0.005 degrees, and a user-supplied GTI is supplied in extra.gti
(which is merged with "AND" merging with the maketime GTI).
SEE ALSO
niprefilter,
maketime
ftadjustgti
LAST MODIFIED
Feb 2024