Search:

[Advanced Search]



NICER Analysis Tips & Caveats

Last Updated: October 2, 2023

Here we list a number of issues of which users should be aware when analyzing NICER data. These include calibration uncertainties, data processing issues, and other issues.

For additional documentation we recommend consulting the following pages:


Contents:

Increase in Daytime Light Loading (May 2023)

Summary

On May 22, 2023, between 13:00 and 14:00 UTC, NICER appears to have developed a visible-light leak in the X-ray Timing Instrument (XTI) optical bench.

The light leak has caused a significant impact to observations, and NICER has implemented several mitigations to reduce these impacts. These impacts include increased undershoots, increased deadtime, and high chances of lost data. Some of the mitigations include reducing the observable field of regard and raising the low energy threshold during orbit day.

These changes will have a significant impact on how you analyze NICER data.

For more details, please see the following pages:

MPU1 Time Stamp Anomaly from July 8th-23nd 2019

From Day of Year (DOY) 189 (July 8) ~14:18 UT till DOY 204 (July 23) 21:21 UT, NICER experienced a time stamp anomaly due to a Single Event Upset (SEU) that occurred during an SAA passage on July 8th. This caused events collected by MPU1 (detectors with DET_IDs from 10 to 17 inclusive) to have incorrect time stamps. As an example, a power spectrum from these detectors is shown in the figure below.

Figure showing the effect of the incorrect timetags from MPU1 on the NICER data

Figure 1. An example showing the effect of the incorrect timetags from MPU1 on the NICER data.


On July 22nd, MPU1 was power cycled, which resolved the time-stamp issue for MPU1 as indicated in the power spectrum shown below. No other MPUs were affected during this time.

Figure showing the effect of the incorrect timetags from MPU1 on the NICER data

Figure 2. An example showing NICER data after MPU1 was reset.

Internally, NICER timestamps are recorded with an oscillator that runs at ~25 MHz. Normally, every 1 second, these timestamps are registered to a GPS clock. The lower 25-bits of the count are stored on a per-event basis, and higher order timestamp bits are transmitted separately. When the upset occurred, the lower 25 bits became scrambled, while the higher order bits are accurate. Thus, the symptoms of the problem are accurate timestamps at the ~1.3 second level, but inaccurate timestamps at finer time resolution.

Observations for which timing of events is critical analysis (e.g. pulsar light curve modeling, QPO analysis, etc...) will be affected if data from DET_IDs 10 through 17 are included in the analysis for observations in the July 8th-22nd 2019 interval. A table of the OBSIDs for the affected observations is given below (and attached Obsids-unique.xlsx). For these observations, if one is doing timing analysis, MPU1 data should be removed from the combined event file using one of the following methods.

The recommended approach is to run nicerl2 with a modified mpulist option:

> nicerl2 ... detlist=launch,-MPU1 ...

where the mpulist above excludes MPU1.

For "quick and dirty" analysis of an existing event file, one can simply remove data from MPU1 using nifpmsel:

> nifpmsel input.evt output.evt detlist=launch,-MPU1

Valid spectra can be extracted from these observations even if the MPU1 detectors are included. Please note that bright sources might have small deadtime errors, so a comparison of spectra with MPU1 included, and with MPU1 excluded, is advisable. Note that MPU1 normally reads out 7 detectors.

A file listing OBSIDs for the NICER observations affected by this anomany is available here.


Low Energy Noise

In general the useful energy band for NICER is 0.2 < E < 12 keV. The lower bound is set by the detector noise, which peaks near 0.1 keV. However, for data taken during orbit day, the low energy noise peak may intrude into the low energy band. Users should examine carefully the band at the lowest NICER energies. The noise peak tends to increase during orbit day, so that users should be careful about interpreting events in the 0.25 – 0.5 keV range for data taken during orbit day. In some cases, data at E ≤ 0.5 keV may need to be excluded from analysis.


Excluding data from noisy detectors

Analysis of detector behavior on-orbit shows that, under certain conditions, two of the NICER focal plane modules, namely DET_ID = 34 and DET_ID = 14, exhibit episodes of increased detector noise. Users may want to exclude data from these detectors in their analysis. This can be done using nifpmsel: For "quick and dirty" analysis of an existing event file, one can simply remove data from these MPUs using nifpmsel: 

> nifpmsel input.evt output.evt detlist=launch,-14,-34
where input.evt is the cleaned events file with data from all 52 FPMs, and output.evt is the output cleaned events file including detectors in operation at launch (which is all detectors except detectors 11, 20, 22, and 60), and excluding FPMs 34 and 14.

Energy Calibration (Gain)

Nominally, the NICER PI detector channels are linearly spaced and 10 eV wide, covering the range 0.0-15.01 keV. The energy boundaries of the PI channels can be found in the EBOUNDS extension of the XTI response matrix file. There remain some residual uncertainties regarding the NICER energy scale (gain scale). Generally speaking, energy assignment is within 5 – 10 eV, except < 1 keV and in the 4 – 7 keV band (energy deviations in the 4 – 7 keV band are generally about 0 – 30 eV). At this time, users should be very careful about claiming red/blue shifts of < 30 eV in the Fe-K range (in particular) based solely on NICER spectra. In-flight testing found no strong dependence of gain on temperature or lighting conditions.


Photon Redistribution (Response Matrix) and Effective Area

Residuals of about 2-3% remain in the response matrix and effective area, with larger features at specific energy ranges. Users may want to include a systematic 15% uncertainty to spectral analysis. See the NICER Calibration Memo 20200202 for more details.


Timing Calibration

After Level 2 calibration, the timing error is ≲ 100 ns. Note that, before calibration, a 1 second absolute timing offset exists due to the behavior of the on-board clock. As of August 2018, the archived data has a mix of calibration levels applied. For reliable absolute timing results, the user must apply most recent NICER software nicerl2 and use the Calibration Database version 20180711 or later. After processing with nicerl2, the correction for this 1-second offset is recorded in the TIMEZERO keyword in the header of event files, GTIs and filter files. The NICER data extractor will automatically adjust time values using TIMEZERO when creating light curves. Users who have developed custom software for timing analysis will need to account for the TIMEZERO keyword offset.


Deadtime

Deadtime is the time when the detector electronics are "busy" so no events can be recorded. All events, including resets, overshoots and forced triggers, create deadtime. Deadtime is recorded on a per-event basis in the UFA event file. Each MPU is parallel. Deadtime is also recorded in the MPU_DEADTIME column of the .mkf file. The "cleaned" event file (as found, for example, in xti/event_cl/niNNNNNNNNNN_0mpu7_cl.evt) excludes < 0.25 keV pulse heights, 5 forced triggers per second per FPM, undershoot resets (dark current), overshoot resets (typically charged particles).

Deadtime is particularly important for bright sources (≥ 20,000 NICER counts/s using 52 FPM). Very bright sources will cause telemetry saturation and a large increase in deadtime. For bright sources, telemetry saturation can be mitigated by using only a subset of the 52 FPMs to observe the source. This will reduce deadtime, but users should be aware that this will also reduce the total effective area of the XTI, so that this reduction in effective area needs to be corrected when deriving source fluxes. To first order, when N FPMs, are used the effective area is simply reduced by a factor N/52 compared to the standard, 52-FPM effective area normally used.


Fragmented Good Time Intervals

For very bright targets, users must be aware that, at high data rates, telemetry saturation will limit data throughput. The practical result of this is that event data can be fragmented into very short (millisecond) duration data segments. The NICER operations team attempts to avoid this condition, but it is not always possible to predict target intensity or respond quickly to intensity changes. As a result some processed datasets have been archived with large numbers (sometimes thousands or more) of very short good time intervals. Large numbers of short-duration good time intervals can make timing analysis challenging, and may result in identification of spurious peaks in the power spectrum which may be mistaken for periodic or quasi-periodic oscillations if care is not taken. In particular, users creating power spectra must properly deal with the window function created by the good time intervals (this is true for unfragmented observations but especially true for heavily fragmented data).

As noted above, for very high count rates (above approximately 20,000 NICER counts/sec), telemetry saturation can occur. Telemetry saturation can result in many very short good data intervals, with similarly short gaps between them. This information is correctly captured in the good time interval (GTI) table present in all NICER event data files (the GTI extension of the FITS binary tables).

This effect is illustrated in the example below, which uses data obtained from the bright black hole transient MAXI 1820+070. The data presented here were take from obsid 1200120107, using the cleaned event file, ni1200120107_0mpu7_cl.evt. Because of the brightness of this source, the GTI extension of this file shows 5912 individual good time intervals.

Figure 1 shows a continuous light curve of a short portion of the data obtained by binning the events with a bin size of 1/512 seconds. The count rate varies from about 20,000 to greater than 50,000 counts/sec. One can see that at high rates there are many short gaps in the data due to telemetry saturation, resulting in a large number of mostly short GTIs, with short gaps between consecutive GTIs. The presence of very short gaps between consecutive GTIs can result in spurious time signatures in the power spectrum due to the gaps alone, and not the X-ray source of interest.

NICER lightcurve of the bright black hole transient MAXI 1820+070 showing data gaps produced by telemetry saturation for obsid 1200120107

Figure 1. An example of the many short gaps that may result in a NICER light curve
of a very bright source due to telemetry saturation.


As a further example of this, we show a histogram of the time intervals between all GTIs in this event file, where the time intervals are simply the time separations between the midpoints of successive GTIs. The distribution is strongly peaked on timescales from 1/10 to 1/100 of a second.

Distribution of the time intervals between GTIs for obsid 1200120107

Figure 2. Distribution of the time intervals between GTIs for obsid 1200120107 emphasizing
the large number of very short good time intervals for this observations.


A power spectral analysis that includes the data gaps will very likely show spurious (non-source) excess power on these timescales. We show two power spectra from these data, one from a light curve which includes time bins within the data gaps, and one which, correctly, only uses continuous GTIs (in this case, bins which are at least 16 seconds long).

Figure 3 shows these two power spectra. The red curve shows the correct result, using only continuous GTIs, and the black curve shows the (incorrect) spectrum which included gaps in the data. One can clearly see excess power above the red curve, and indeed, a spurious "QPO" near 55 Hz. For timing studies of bright sources, the NICER team strongly recommends double checking the GTIs for each data file.

Power spectrum showing the influence of data gaps on timing analysis

Figure 3. Black: power spectrum of the NICER obsid 1200120107 with data gaps included;
Red: power spectrum of the NICER obsid 1200120107 with data gaps excluded.
Note the spurious peak which is seen in the power spectrum due to short gaps
between the good time intervals.


Pointing Jitter and Spurious Timing Signatures

NICER's X-ray concentrators were co-aligned to within 1 arcmin in order to optimize the X-ray throughput. Each concentrator optic is described by a response function that falls with off-axis angle. Additionally, each optic focuses X-rays onto a detector which has a 2 millimeter-diameter aperture. The combination of each optic's off-axis response and the fixed detector aperture sets the total throughput as a function of the off-axis pointing angle to a target. At about 3 arcmin off-axis the response is reduced by half, mostly due to the aperture stop. For smaller off-axis angles the response is only slightly reduced; for example, within 1 arcmin the response drops by only 2-4%.

NICER points by aiming its "boresight," a fiducial vector specified in the frame of reference of the NICER instrument that maximizes the X-ray throughput, to commanded target positions in the sky. NICER's pointing system is designed to maintain pointing stability to targets of interest at the 1 arcmin level. The summed response of all 52 X-ray concentrators and detectors is approximately "flat-topped" such that angular deviations associated with pointing system jitter at this level are not sufficient to introduce spurious, non-source variability into the observed X-ray flux. This is true for targets with sky coordinates known to high precision (much better than ~1 arcmin), and the NICER team is not aware of any such cases where substantial spurious variability may have resulted from pointing system jitter.

However, for newly identified targets with less precise coordinates (positional uncertainties larger than the arcmin level) users should be aware of the possibility of pointing-related variability being introduced into the observed count rates. The issue here is that when the target coordinates are not precise enough, the source may no longer be sampling the "flat-topped," on-axis response. In such cases, the off-axis variations due to pointing jitter can introduce detectable variations in the flux. A recent example of this effect was seen in observations of MAXI J1348-630 from February 8, 2019 (ObsID 1200530107). Relatively strong (~10% rms) count rate modulations were seen with a quasiperiod near 0.2 Hz. It was determined that for these observations NICER was pointing about 2 arcmin away from the subsequently derived (through independent imaging observations) correct source position. The feature disappeared once the correct pointing coordinates were adopted. The observed 0.2 Hz frequency is a known mechanical resonance associated with the NICER pointing system. The brighter the target, the easier it will be to detect such modulations.

Thus, users should be very cautious when interpreting apparent low-frequency (below 0.5 Hz) variability for bright targets where accurate, sub-arcmin coordinates may not yet be available.

Angular offset (arcmin) from the true source coordinates

Figure 4. The power spectrum of the angular offset (arcmin) from the true source coordinates for observation 1200530107 of MAXI J1348. For this observation, the pointing position is offset from the actual source coordinates by approximately 2.15 arcmin. This power spectrum shows a strong signal near 0.2 Hz (a period of ~5 seconds), which is a known mechanical resonance associated with the NICER pointing system. In mispointed observations like this, the off-axis variations due to the pointing jitter can introduce detectable variations in the observed count rate, producing a spurious (non-source) QPO at 0.2 Hz in the observed count rate. Note that other timing peaks are also seen in the power spectrum.

South Atlantic Anomaly

Particles in the region near the South Atlantic Anomaly (the SAA) produce non-cosmic background which can interfere with interpretations of data. Particle event rates can be very high and swamp source X-ray events. NICER processing uses a boundary defined by a parameter called NICER_SAA which is recorded in the .mkf file. Normally data are excluded from the cleaned event list when NICER_SAA = 1. For analysis of weak sources, or any analysis which is particularly sensitive to particle background, users might instead filter times based on the value of the SAA column in the .mkf file, since the SAA boundary region is considerably larger than the NICER_SAA region. Use of the SAA region for filtering will result in excluding more data than filtering with NICER_SAA, and it may be that some of the excluded data includes valid science events, so care should be taken when filter for times near SAA.


Artificial Dips in Source Count Rates due to Obstruction by Structures on the ISS

Since NICER is an attached payload on the International Space Station (ISS), on rare occasions ISS hardware near NICER can obstruct the NICER X-ray Timing Instrument (XTI) view of the celestial targets. This will cause artificial variability which could be misconstrued as intrinsic variability of the target of the observation. Users should check for blockages before interpreting real, short (on the order of seconds to minutes) dips in X-ray lightcurves. For more information, see the thread on how to determine if ISS structures are temporarily blocking portions of the XTI field of view.

NICER Annealing Times

Since NICER has been on orbit, radiation damage has slowly increased the dark current of its Silicon Drift Detectors (SDDs). The increased dark current has not yet degraded NICER's sensitivity, but since the radiation-imduced dark current increases with time, we have begun exploring ways to reduce dark current. Radiation damage in SDDs is produced by trapped protons in the South Atlantic Anomaly (SAA), and manifests primarily in the form of dislocations within the crystal structure of the SDD's silicon. It is commonly known that annealing of these devices at elevated temperatures has the effect of moving the dislocations toward the crystal boundaries, where they disappear. In nominal operation, NICER's SDDs are individually cooled to -55 C using thermoelectric coolers (TECs). Turning off an SDD's TEC will cause it to warm to about 0-5 C during normal NICER operations.

Therefore, starting in mid-2020, the NICER project has begun an annealing program that heats one SDD at a time by turning off the SDD's TEC for about 1 week (while all the others continue operating). This results in a decrease in NICER's effective area of less than 2%. During that week, some of the radiation damage suffered by that device will be annealed. The following week, the annealed SDD is brought back on-line while another SDD is annealed. Thus, in 1 year, each SDD will get about 1 week of elevated (0-5 C) annealing. Initial evidence shows that 1 week of annealing should remove 6-9 months of radiation damage-induced dark current.

The HK parameter FPM_UNDERONLY_COUNT for one detector is shown as a function of spacecraft clock time. Data are limited to orbit night to maximize our sensitivity to thermal dark current. The drop in FPM_UNDERONLY_COUNT is due to a 5-day annealing test. After that test, the SDD's dark current returned to the lower level it had roughly 6 months earlier.


Figure 1: The HK parameter FPM_UNDERONLY_COUNT for one detector is shown as a function of spacecraft clock time. Data are limited to orbit night to maximize our sensitivity to thermal dark current. The drop in FPM_UNDERONLY_COUNT is due to a 5-day annealing test. After that test, the SDD's dark current returned to the lower level it had roughly 6 months earlier.

Users of NICER data should be aware of a possible decrease in effective area due to annealing in observations obtained after July 21, 2020. Please see the list of annealing times to for information on when a particular detector is inactive due to annealing. Users are also advised to extract lightcurves of individual detectors to look for unexpected drops in detector count rate caused by annealing, ISS obstruction or other detector issues.

Standard Processing Results in No Clean Events

Standard processing accepts good science times based on a number of criteria. For some observations, the standard criteria may result in zero good exposure and thus no accepted X-ray events. In particular, NICER processing screens out times of contamination due to solar optical light leakage and also times when the high energy particle rate is high. Relaxing these criteria may increase the number of X-ray events. Solar light leaks cause increased number of "undershoots" (detector resets triggered by accumulated charge) and times of high undershoot rate which are usually rejected by standard processing. The undershoot rate is given by the FPM_UNDERONLY_COUNTS in the .mkf file. For the standard NICER processing the undershoot limit is set to 200 counts/s. At low sun angles (SUN_ANGLE < 70 degrees) the undershoot rate may exceed this limit, even though the excluded time may include good X-ray photons. Reprocessing your data with nicerl2 with a higher undershoot limit (up to 400 cts/s) may yield non-zero exposure and X-ray photons. For example running nicerl2 on a NICER <obsid> as below: 

% nicerl2 <obsid> underonly_range=0-400
would include as good science time those times when the undershoot rate is in the 0-400 range (compared to the 0-200 range used for standard processing). Increasing the undershoot limit may recover more real X-ray events. However, increasing this limit may affect the gain calibration. Note that observations at low sun angles will also increase the noise at the lowest energies due to optical loading. Depending on the amount of optical loading, you may want to ignore ignore PI < 40 or 50, but examine your spectra to determine the appropriate PI restriction. Standard processing also excludes times of high energy particle rates (the "overshoot" rate, as measured by the FPM_OVERONLY_COUNT column in the .mkf file). Sometimes the standard constraint can be too restrictive, yielding no accepted X-ray events. To include times with a higher overshoot rate as valid science times, you can run nicerl2 as in the example below: 
% nicerl2 <obsid> overonly_range=0-2 overonly_expr="2.0"
Note that increasing the overshoot limit might increase the amount of background in your observation, especially at high energies.

Bug in HEASoft 6.27 and 6.27.1 (ERROR: could not find MKF file)

NICER scientists should be aware that HEASoft versions 6.27 and 6.27.1 (which were released on March 26, 2020 and April 13, 2020, respectively) contain a bug that will cause nicerl2 to finish with an error (ERROR: could not find MKF file). The first time nicerl2 is run, no error is produced, but if a user attempts to run nicerl2 a second time, the error occurs.

The error is ultimately due to a bug in niprefilter2. The OBS_ID keyword is changed from a string-type keyword to an integer-type keyword, which nicerl2 cannot handle properly.

The NICER GOF recommends that all users of HEASoft 6.27 and 6.27.1 must upgrade to HEASoft 6.27.2, released on April 23, 2020. This patch release will correct the error. In addition, for those files where the OBS_ID keyword has been altered, the new patch release version of nicerl2 will correct the keyword data type of OBS_ID.

Absolute 1 second timing errors with extractor 5.37 or earlier

The NICER team has become aware of an absolute timing error for event files extracted by the tool extractor (version 5.37 or earlier), as well as xselect. The versions of HEASoft that are affected by this problem are 6.29 or earlier.

Please see the NICER Time page for more details.

NICER Response Bug Fixes (Weighting and TIMEZERO)

There were two bugs in NICER response calculators distributed in NICERDAS 8, 8a and 8b (distributed with HEASoft releases 6.29, 6.29a, and 6.29b in August 2021). The two bugs are: 1) For most cases, detector weighting in FPM Selection Data is ignored, leading to incorrect response calculation, and 2) a 1-second time offset leads to incorrect response calculations, which is magnified for "shredded GTIs" (and could lead to 10-30% error). These bugs were fixed in NICERDAS 8c distributed with HEASoft 6.29c. See the NICER Response Bug Fixes (Weighting and TIMEZERO) analysis thread for details.

Common errors and how to resolve them

Users may find the analysis thread "Common NICER Error Messages and How to Fix Them" useful in resolving analysis issues.

Spurious Timing Signal for NICER MPU 1 and 3

Since (at least) the start of 2023, low-sun angle data sets show non-science-related (spurious) power in a narrow band at 55 Hz, and a broad feature in the 18-30 Hz range. This timing feature is seen only on MPUs 1 & 3 (zero indexed), approximately identical for both MPUs and independent of energy. The NICER team is currently investigating the cause of this signal, and any remedies. Users should exercise caution in interpreting signals near these frequencies.