The task xtdflagpix flags Xtend events that may be unsuitable to use in data analysis, such as those that fall on defective or non-sensitive pixels, those that might have missing or compromised telemetry, and those for which processing has encountered certain errors. The STATUS column in the input events file is updated, encoding the flagging condition for each event. This tool does not actually filter or remove events. However, the STATUS column can be used for later filtering. An image in DET coordinates showing the location of unsuitable pixels is also output for later use in constructing an ARF or flat field image.
The tool uses several input files to assign STATUS: a list of bad or dead pixels or columns that don't change ('badpixfile'); a list of hot pixels, which can change on short timescales and are observation-specific ('hotpixfile'); a list of flickering pixels, which can be produced by the task searchflickpix ('flickpixfile'); and a mask file that encodes the location of CCD boundaries, segments, calibration source regions, and other regions that don't change ('maskfile'). The locations of the charge injection rows are read from the input event list header, as are settings for window mode and area discrimination (non-active areas of the CCD).
In addition to an output event list with updated STATUS column, the xtdflagpix task optionally outputs a bad pixel list (specified by the parameter 'outbadpix') containing the events that have any STATUS flag set by the task. Also output is an image (specified by the parameter 'outbadimg') in Xtend DET coordinates containing a value for each pixel corresponding to its STATUS. The values in these files are, in order of increasing precedence:
0: a good pixel 1: a pixel inside the calibration source region 2: a bad pixel set by the parameter 'bad_status' -1: a pixel out of the CCD, window, or area discrimination region
For example, a pixel inside the calibration region but flagged as bad has value 2 in the output image. Flickering pixels are not included in this map because they can depend on time. Note that the output bad pixel list contains only those pixels for which events are found in the input file, while the output image contains all pixels, including those without events, since the image is used to determine the good area of the Xtend field of view.
Below is a table mapping flag position in the STATUS column to particular flagging conditions. If a STATUS flag is set to 1, the condition associated with that is considered to be flagged. Unless otherwise noted, the STATUS values apply to the central pixel of an event. Tools other than xtdflagpix that set STATUS values are noted in parentheses. CI indicates charge injection.
=========================================================================== STATUS Description flag =========================================================================== 1 Any of the flags specified in 'bad_status' are set; a bad event 2 Inside the calibration source region --------------------------------------------------------------------------- 3 Out of CCD (Out of area) 4 Out of window 5 Out of area discrimination --------------------------------------------------------------------------- 6 CI row (Pixels) 7 Bad pixel from CalDB 8 Bad column from CalDB 9 Hot pixel from pre-pipeline 10 Flickering pixel --------------------------------------------------------------------------- 11 CCD boundary (Boundaries) 12 Window boundary 13 Segment boundary 14 Area discrimination boundary 15 At least one 3x3 surrounding pixel has a bad status --------------------------------------------------------------------------- 16 CI trailing row in rows specified by 'citrailnbr' (Neighbors) 17 CI preceding row in rows specified by 'ciprenbr' 18 Preceding/following pixel in a partial bad column 19 Neighbors of bad/hot pixel and bad column 20 Neighbors of flickering pixel 21 Neighbors preceding/following a bad column 22 Neighbors of CCD/window boundary 23 Neighbors of segment boundary --------------------------------------------------------------------------- 24 (xtdphas) 3x3 info is present, but 5x5 is absent (Others) 25 (xtdphas) 3x3 is absent 26 (xtdpi - general) PHAS[0] < event threshold 27 (xtdpi - vtevnodd) Video temperature is out of range 28 (xtdpi - vtevnodd) Lack of video temp HK at time close to the event 29 (xtdpi - chtrail/CTI) Correction value is negative 30 (xtdpi - general) Null value by correction process --------------------------------------------------------------------------- 31 1st trailing row of the CI rows (Diagnostics) 32 1st preceding row of the CI rows 33 2nd trailing row of the CI rows 34 2nd preceding row of the CI rows 35 3rd trailing row of the CI rows 36 3rd preceding row of the CI rows --------------------------------------------------------------------------- 37 Cosmic ray echo pixel (Cosmic Rays) --------------------------------------------------------------------------- 38-48 Reserved ===========================================================================
The special STATUS flag 1 is set to 1 for an event that has any STATUS flag set out of the list given by the 'bad_status' parameter. This is an OR'd list, so any flag listed in 'bad_status' that is set will cause STATUS flag 1 to be set as well. Such events are considered bad, and this facilitates easy filtering later in the processing, allowing good events to be extracted with a single filter expression like "STATUS[1]==b0".
Several STATUS conditions apply to events where neighboring pixels are flagged. Because an event is composed of a 3x3 pixel island, the quality of the region immediately surrounding the event center must be taken into account. For example, a bad pixel in the outer ring of a 3x3 island does not produce a valid pulse height, and this could result in an event with an otherwise unacceptable grade being accepted as a good event. The parameters 'npixnbr' and 'nboundnbr' can be used to set the maximum distance away from a bad pixel or boundary to flag further pixels and events. In addition, for more flexibility, xtdflagpix writes a new column, PHAS_MASK, to the output event list. This is a 9-element mask of the PHAS column with each element indicating whether the corresponding pixel i in PHAS is good (PHAS_MASK[i]=0) or bad (PHAS_MASK[i]=1). PHAS_MASK can be used later in the xtdpi task to determine how to grade the event and calculate the summed PHA using the xtdpi 'badpixopt' parameter. Since xtdpi can overwrite the PHAS column, the ability to reset it to the original telemetry values is provided here with the 'copyphas' parameter. Similarly, 'resetflags' can be used to reset all the xtdflagpix STATUS flags and the elements of PHAS_MASK to zero. Using 'resetflags=yes' does not change the flags shown above that are set by xtdpi or xtdphas.
Some pixels are affected by cosmic ray echo, in which a cosmic ray detected during an Xtend dark frame can cause an echo in a different readout segment. This echo then causes those pixels to have a pulse height above the split threshold for the duration of the observation, leading to incorrect grades for events centered in neighboring pixels. This can be corrected for by flagging such events using the 'echoflag' parameter, which is a comma-separated list of values specifying which CCD segments to search for such pixels, with "1" turning on the flagging, and "0" turning it off. If at least a fraction ('echofrac') of the events containing a given pixel have a pulse height greater than or equal to 'echospth' in that pixel, then the pixel is considered a cosmic ray echo pixel. Any events centered 'echonbr' pixels from such pixels have STATUS flag 37 set, and if flag 37 is specified in 'bad_status', any pixels within 'echonbr' of that pixel will be marked as bad in 'outbadimg'. For Xtend, 'echonbr=2' is the recommended value, which will flag a 5x5 area of pixels around any cosmic ray echo pixel. Only pixels that have data from at least 'echomin' events are considered. An image of the echo fraction can be written out as an extension to 'outbadimg' if this flagging is performed for any segment. This image can be used for diagnostic purposes.
Charge trailed or leaked from charge injection rows can produce spurious events in several rows immediately surrounding the injected rows, manifesting as noise at low energies. The parameters 'citrailnbr' and 'ciprenbr' can be used to tune a compromise between lower noise and higher efficiency, since flagging additional rows reduces the effective area. These will set flags 16 and 17 for affected pixels. Flags 31 to 36 can be used to flag specific CI trailing and preceding rows rather than a range, and are for diagnostic purposes only.
Some examples of how to use the STATUS flags in filtering and extracting events are shown below (see examples section). See the help file for calc_express for proper filtering syntax.
xtdflagpix infile=xa100050020xtd_p0100004b0_uf.evt outfile=output.evt outbadimg=outbadimg.fits \
hotpixfile=xa100050020xtd_a0100004b0.hpix flickpixfile=xa100050020xtd_a0100004b0.fpix
xtdflagpix infile=xa100050020xtd_p0100004b0_uf.evt outfile=output.evt outbadimg=outbadimg.fits \
hotpixfile=xa100050020xtd_a0100004b0.hpix flickpixfile=NONE badpixfile=NONE
xtdflagpix infile=xa100050020xtd_p0100004b0_uf.evt outfile=output.evt outbadimg=outbadimg.fits \
hotpixfile=xa100050020xtd_a0100004b0.hpix flickpixfile=xa100050020xtd_a0100004b0.fpix \
citrailnbr=4 ciprenbr=2
xtdflagpix infile=xa100050020xtd_p0100004b0_uf.evt outfile=output.evt outbadimg=outbadimg.fits \
hotpixfile=xa100050020xtd_a0100004b0.hpix flickpixfile=xa100050020xtd_a0100004b0.fpix \
bad_status=3:9,11:12,16
ftcopy output.evt[events][STATUS[1]==b0] output_filtered.evt
ftcopy output.evt[events][STATUS[1]==b0 && STATUS[2]==b1] output_filtered.evtAnother way to do the same filtering is to use the bxxx bit mask notation, which is explained fully under the help for calc_express:
ftcopy \
output.evt[events][STATUS==b01xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] \
output_filtered.evt
The wild card "x" indicates any value is allowed.
searchflickpix, xtdphas, xtdpi, calc_express, xaexpmap
February 12, 2024