LCSTATS (Jan96) xanadu.xronos LCSTATS (Jan96)
NAME
lcstats -- calculates statistical variables for 1 input time
series and prints on the screen the results
USAGE
lcstats file(s)+options window dtnb nbint
DESCRIPTION
This task performs statistical analysis for one times series and
prints the results on the screen (no output is produced). The input
file format is FITS using the BINTABLE extension. Both binned data
format and event format are input. Data can be rebinned and divided
into Intervals (See GENERAL XRONOS TERMINOLOGY). Time, Phase,
Intensity and Exposure windows (See WINDOW) allow for data
screening. The quantities calculated are : newbin integration time;
interval duration; number of good accepted newbins in interval;
average (and its error); standard deviation; minimum and maximum
count rate per interval; variance (amnd its Gaussian error)
evaluated from the data scatter; expected variance from a constant
source (and its Gaussian error), as evaluated from the errors in
the newbin count rate; third moment; average absolute deviation;
skewness (and its Gaussian error); RMS fractional variation (and
its Gaussian error) or a 3 sigma upper limit if the variance is not
larger than the expected variance at a confidence level higher than
99.86%; Chi-square and the corresponding number of degrees of
freedom; constant source probability associated to the Chi-square
value; constant source probability from a Kolmogorov-Smirnov test.
GENERAL XRONOS TERMINOLOGY
Within XRONOS tasks, BINS and NEWBINS control the binning used in
the analysis, INTERVALS the subdivision of the time series and
FRAME the grouping of the output results:
BINS : these are the time bins of the time series being analysed.
More than one input file can have different bin durations, e.g. two
consecutive time series, one with 0.5 s bins and the other with 2 s
bins. The original bin time is the value stored in the input file
in the keyword TIMEDEL. If the data are stored in each row as an
array with 1CTYPn = 'TIME', the original bin is set to the value
stored in the keyword 1CDLTn (where n is the column number).
NEWBINS : these correspond to the time resolution at which the
analysis is carried out. Note that: (i) newbins cannot be shorter
than the longest bin duration of the time series being analysed;
(ii) in many XRONOS applications (e.g. powspec, autocor, crosscorr)
the newbin duration is forced to be an integer multiple of the
longest bin duration.
INTERVAL : an interval is defined by the number of newbins over
which the analysis is carried out. Note that in applications using
FFT algorithms (e.g. powspec, autocor and crosscor set in fast
mode) the number of newbins in an interval is a power of 2.
FRAME : a frame consists of the average of the results of the
analysis of one or more contiguous intervals. Note that in
'lcurve', 'efsearch' and 'lcstats' a frame consists always of one
interval.
WINDOWS
If any window is required during the analysis, a window file
containing the relevant windows must be created with the
application XRONWIN, before running a XRONOS task. There are 4
different types of windows :
* Time Windows : consist of up to 1000 time intervals
* Phase Windows : consist of an Epoch, a Period and up to 10 phase intervals
* Intensity Windows : consist of up to 10 intensity in bin, newbin and interval
* Exposure Windows : consist of up to 1 exposure in bin, newbin and interval
Intensity and Exposure Windows can be specified independently for:
(i) Bins , (ii) New Bins , (iii) Intervals. When dealing with more
than one time series, Intensity and Exposure Windows must be
specified separately for each series. Time and Phase windows are
applied to Bins. Intensity and Exposure windows are applied first
to Bins, then Newbins and finally to Intervals as specified. For
time and phase windows, only those bins whose center time is within
the start and stop of a time window or phase window (for a
specified epoch and period) are accepted. Intensity windows must be
ordered with increasing intensity and if set for newbins can be
used in conjunction with "Special Newbin Windows" (see below).
Exposure Windows consist of a minimum and a maximum exposure level.
Units are such that 1 means 100% exposure. The Newbin Exposure is
obtained by propagating the bin exposures to each newbin. For
example, if in a 30 s newbin the total exposure (due to the sum of
the individual exposure of the bins contributing to the given
newbin) is 18 s then its exposure is 60%. The Interval Exposure is
the ratio of accepted to expected newbins: for example, if a 128
newbin long interval contains only 32 accepted newbins, then its
exposure is 25%. Many XRONOS application use some default exposure
windows, which are designed to avoid analysing data sets which are
too inhomogeneous with respect to their statistical properties. The
minimum default Exposure windows in an Interval is set to 0.0 in
the lcurve, efold and efsearch and to 0.5 (i.e. 50% exposure ) in
all the other tasks. Note that exposures can be higher than 100%
(e.g. if the newbin time is not a multiple of the bin time, then
"beats" are generated which might bring the exposure of a newbin to
values >100%; or if two or more input files for the same time
series overlap in part, some of the newbins will be more than 100%
exposed). IMPORTANT NOTE WHEN TIME WINDOWS ARE SET IN THE WINDOW
FILE: The time used within XRONOS tasks is Truncated Julian Days
(TJD= JD-2440000.5) if either (1) the keyword MJDREF is present in
the header or (2) if the TIMESYS value is one of the following
strings MJD or JD or TJD. If (2), the time values are expected to
be stored as JD, MJD or TJD in the header keywords and in the TIME
column in which case the MJDREF keyword is not used (it should not
be present). When Time windows are set using XRONWIN, they must be
compatible with the values in header of the timing keywords and/or
the values in the TIME column.
An additional window type called "Special Newbin Window" can be set
directly from the parameter file. Special Newbin Windows are used
to exclude the parts of a light curve which immediately follow or
precede a burst or a background event which has been rejected by
intensity windows in newbins. The Special Window operates on
newbins in conjunction with intensity windows (in newbins) and are
specified by changing to positive values the parameters
'spwinbefore' and 'spwindowafter'. Their use is the following: if
e.g. spwinbefore is set =10, all newbins, whose center time is
within 10 second before the center time of a newbin rejected by
intensity windows, will also be rejected; if e.g. spwindowafter is
set =20, all newbins, whose center time is within 20 second after
the center time of a newbin rejected by intensity windows, will
also be rejected.
FILELIST and INPUT FILE OPTIONS
To input multiple files for each time series, a file containing the
list of files is needed (Filelist). The Filelist is input in the
program as '@Filelist'. The format of this file list is ascii and
contains one filename+options per line. Files from different times
series are separated by '///' mark. Below is an example of the
Filelist containing 2 files for 3 different times series.
file1_ser1
file2_ser1
///
file1_ser2
file2_ser2
///
file1_ser3
file2_ser3
The Input File Options (up to 10) can be specified for each file in
the same input string. They consist of 2 characters followed by a
numerical constant (up to 8 character long). There are two groups
of options. The first allows data selection within a FITS
extension. The available options within this group are :
frN= start reading input file from row number N (first row)
lrN= stop reading input file from row number N (last row)
vxN= use column number N as x-axis (i.e. time axis, default name is TIME)
vyN= use column number N as y-axis (default names are COUNT or RATE)
vsN= use column number N as error for y-axis (default name is ERROR)
veN= use column number N as exposure (default name FRACEXP).
If the input file is an event list, exposure is by default
calculated using the GTI extension. In this case, N=0
turns off the usage of the GTI extension for the exposure
calculation, and N > 0 specifies the GTI extension to use.
feN= select data (either binned or events) from channel number N (First Energy).
For an event list channel selection is made using the column named 'PHA'
leN= select data (either binned or events) to channel number N (Last Energy).
For event list the default column channel name searched is 'PHA'.
The option 'vcN' allows the choice of a channel column name different from 'PHA' (es. 'PI').
vcN= use column number N for channel selection (valid only for event lists).
rtN= use extension N of the FITS file to read the data. The first extension
is N=1 (the primary array is irrelevant). To specify the extension the
following also can be used: filename[N] or filename+N.
of = The MJDREF keyword is not used. The time is calculated using the
TIME column and the TIMEZERO keyword.
The second group of options performs algebraic operations on
individual input files. They are applied in the same order in
which are specified. For event files they are applied after the
data are binned. The available options within this group are:
stX = Shift all Time in input file by X days
ssX = Shift all times in input file by X Seconds
muX= multiply data and errors by X (MUltiply)
mdX= multiply data by X (Multiply Data)
meX= multiply errors by X (Multiply Errors
maX= as muX but exposure is divided by X
diX= divide data and errors by X (DIvide)
ddX= divide data by X (Divide Data)
deX= divide errors by X (Divide Errors)
daX= as diX but exposure is multiplied by X
aaX= add data and errors with X (Add All)
adX= add data with X (Add Data)
aeX= add errors with X (Add Errors)
saX= subtract data and errors with X (Subtract All)
sdX= subtract data with X (Subtract Data)
seX= subtract errors with X (Subtract Errors)
qaX= add to data the square of data muliplied by X and add to errors
the product of data and error multiplied by X
qdX= as above but for data only
qeX= as above but for error only
Below is an example of the Filelist containing 2 files for 3
different times series where the different options are applied to
the input files for different time series.
file1_ser1 aa4 add to data and error 4
file2_ser1 aa4 " " " "
///
file1_ser2 rt2 aa2 read 2nd extension; add to data and error 2
file2_ser2 rt2 aa2 " " " " " " "
///
file1_ser3 rt2 vy4 vs5 read 2nd extension; use column 4 and 5 for Y-axis and Error
file2_ser3 rt2 vy4 vs5 " " " " " " " " " "
PARAMETERS
cfile1 (filename(s) first series+options) [string]
Input filename(s) for the first time series + options. The
valid input files are in FITS format using the BINTABLE
extension. Xronos tasks read for each time series many
consecutive input files (up to 50). Additional flexibility is
provided by Input File Options which are used to perform
algebraic operations on individual input files (either on the
'times' or on the 'count' or 'count/s' values). The Input file
Options are also used to select columns and rows within a FITS
file. If the first character of the input string is '@', the
rest of the string is taken to be a filename containing the
list of input files (Filelist). The Filelist can contain
filenames for more than one series. See description of
"FILELIST and INPUT FILE OPTIONS".
window (name of window file) [string]
Filename of the xronos window file. The window file is an ASCII
file and by default a standard window file is used, where only
exposure windows are set. To modify the standard file or
create a new file used the script XRONWIN.
dtnb (integration time) [double]
The duration in seconds of the NEWBIN time. For binned input
files the NEWBIN duration can not be shorter than the longest
bin duration in the input file. In a number of XRONOS tasks the
NEWBIN time must be an integer multiple of the minimum newbin
time. The task internally calculates (and prints on the
screen) a default value such that a single interval is produced
with a fixed number of newbins (typically between 128 and 4096
depending on the task and on the time interval length see also
"nbdf" parameter). Typing 'INDEF' forces the task to use as
NEWBIN time the value calculated by the program. NOTE: By
pressing return the task will use as NEWBIN time the value
found in the parameter file used in a previous run.
nbint (number of points per interval) [integer]
The number of newbins per interval used in the analysis. The
"nbint" together with the NEWBIN duration determines the length
in time of an interval and therefore the total number of
intervals within the start and stop time over which the
analysis will be carried out. Typing 'INDEF' forces the task
to use the default value (see parameter "nbdf"). NOTE: By
pressing return "nbint" is set to the value found in the
parameter file used in a previous run.
itre (Flag for trend removal) [integer]
A polynomial trend, up to 4th-order, can be removed from input
time series. Setting the parameter "itre" equal to 1 or 2 or 3
or 4 remove a 1st, 2nd, 3rd, 4th-order polynomial trend,
respectively. The trend is determined separately for each
interval of each series by using a least-square technique. The
value 0 does not cause the removal of any trend from the input
series and is the default value. NOTE The trend removal is not
available for the efold and efsearch tasks.
itremo (Mode for trend removal) [integer]
Specify how the trend removal is applied to the data (available
only id "itre" is higher than 0). The trend can be subtracted
from the time series (itremo =1) , or the time series can be
divided by the trend (itremo = 2), or the time series can be
replaced with the trend (itremo = 3). By default the trend is
subtracted (value set to 1).
tchat (terminal chattiness) [integer]
Set terminal chattiness: (0-4) only little information is
output in running XRONOS task ; chattiness 5 is the default
value; (6-7) more details on input files, windows, intervals
statistics, etc.; (<8) mostly for debugging purposes.
lchat (log file chattiness) [integer]
Set log file and chattiness in the log file: = 0 the log file
is not written; for all other values, information is written in
the log file. The chattiness levels are the same as for the
terminal.
logname (log filename) [string]
Name for the log file. The default name is xronos.log.
clobber [boolean]
Flag specifying whether or not a pre-existing file with the
same name as that requested for an output file in the current
task will be overwritten. Default value = yes.
(dpath = XRDEFAULTS) [string]
This string parameter gives the path to the Xronos 'defaults'
directory, which contains the default '.pco' file (used for
plotting) and the defaults window file 'default_win.wi'.
Ordinarily, the user may leave this parameter set to the string
'XRDEFAULTS', which causes Xronos to use the environment
variable XRDEFAULTS to locate these files. XRDEFAULTS is set by
the mkftools script to point to the appropriate directory for
the current distribution of Xronos (for FTOOLS v3.6 this is
/ftools/xronos/defaults/). If the user wishes to modify these
files, he or she may make and edit copies, and change the
XRDEFAULTS variable appropriately using setenv, but the
original files should not be changed.
gapfill (running mean gap filling) [integer]
Replace gaps in input series with running mean. If =0 (the
default) data gaps are not filled. If =n newbin data gaps
in input series are filled in with running mean values
calculated over n newbins. Note that a gap newbin is filled
in only if the corresponding running mean is calculated
over n/4 points at least (this means that in order to
bridge a gap of m newbins n must be >1.35m). This global
parameter is ignored in epoch folding applications (efold
and efsearch).
forcestart (flag for start time) [boolean]
If = yes the first interval will be forced to start at the
time of the first time window otherwise (=no default) the
center time of the first qualified newbin is used as the
start time.
errorbars (Error bar Evaluation) [integer]
This parameter defines the way in which the error bars of
the analysis results are calculated. If the number of the
intervals per frame ("nintfm") is higher than "errorbars"
value (default=5), the error bars are evaluated by using
the standard deviation of the average (based on the
measured scatter). Otherwise the error bars are evaluated
by propagating the theoretical error bars through an
averaging process. Note that for several XRONOS
applications (e.g. `autocor`, `crosscor`) only the former
way of evaluating error bars is available. For example, if
"errorbars" is 5 in the application `powspec`: (a) if a
frame contains the average of 5 or fewer power spectra,
then the error bars in the average power spectrum will be
calculated by propagating through the average the
theoretical error bars associated with each power spectrum
(in turn obtained from the relevant chi-square
distribution); (b) if a frame contains the average of 6 or
more power spectra, then the error bars in the average
power spectrum will be calculated by evaluating the
standard deviation of the average power for each frequency.
By adjusting the value of "errorbars" it is possible, e.g.
to evaluate error bars as in (a), also in the case in which
a large number of intervals per frame has been specified.
Values < 5 are not recommended (at least 5-6 measures are
necessary to reliably evaluate the standard deviation of
the average from the scatter around it). NOTE: not
applicable for 'lcurve', 'lcstats' and 'efsearch'.
exposure (flag for analysis of exposure profile) [boolean]
If =yes the exposure profile(s) (i.e. newbin values are set
=1, gaps and rejected newbins are set =0) is/are analysed
(instead of the input series). Default value is = no.
normalization (type of normalization) [integer]
Flag to specify the type of normalization to apply to the
results. This parameter is only relevant for the following
tasks: powspec, autocor, efold, crosscor, timeskew). The
standard normalization corresponds to a value of 1 (the
default) in all XRONOS applications. Other normalization
value flags are described for each application (See
normalization). NOTE: not applicable for 'lcurve' and
'efsearch'.
simultaneous (flag for simultaneity) [boolean]
If =yes a strict simultaneity is forced between the input
series in applications which use more then one series (i.e.
if the n-th newbin of a series is a gap or is rejected,
then the n-th newbin of all other series will be also
rejected). This flag is ignored in the efold applications.
Default value is = no.
spwinbefore (special window start) [double]
Special newbin window : number of seconds before. If a
value > 0 is used , e.g. 10.0, then all the newbins within
10.0 seconds before a newbin rejected by an intensity
window will also be rejected. The default (=0) is not to
apply this type of special newbin window.
spwinafter (special window stop) [double]
Special newbin window : number of seconds after. If a value
>0 is used , e.g. 10.0, then all the newbins within 10.0
seconds after a newbin rejected by an intensity window will
also be rejected. The default (=0) is not to apply this
type of special newbin window.
rescale (rescaling for results) [double]
Rescaling factor applied to result variables and errors.
The rescaling is applied just before writing the output
file (this to avoid affecting the statistical variables for
the frame). Default value for "rescale" is set to 1.
offset (additive constant for results) [double]
Additive constant summed to result variables. Result error
bars are left unchanged. The additive constant is added
just before writing the output file (this is to avoid
affecting the statistical variables for the frame). Note
that if a rescaling factor is also specified (different
from 1), then the results are first multiplied by the
rescaling factor. Default value for "rescale" is set to 0.
fast (Flag for fast algorithm) [boolean]
This parameter sets the type of algorithm used for the
Fourier transform. IMPORTANT NOTE: This parameter can be
set by the user only in the `powspec`, `autocor` and
`crosscor` tasks (it is a query parameter in these tasks).
In all the other tasks "fast" is a non-query parameter, and
should not be changed from the default setting.
ipow2 (Flag if power of 2) [integer]
Internal Flag used to decide if the current task must used
with a power of 2 of number of points (ipow2=1) per
interval or not (ipow2=0). IMPORTANT NOTE: This parameter
should not be changed by the user.
iavgreb (Flag if average interval) [integer]
Internal Flag used to decide if the current task allows the
averaging of intervals in frame and/or the rebinning of the
analysis results. If "iavgreb" is set to -1 the results in
an interval can not be either averaged or rebinned, if set
to -2, the results in an interval can be averaged but not
rebinned. IMPORTANT NOTE: This parameter should not be
changed by the user.
nbdf (Default No. Bins) [integer]
Set an internal default value for the number of newbins per
Interval. This value is used to calculate the default
newbin integration time to have one interval with nbdf
points. Different "ndbf" values have been set for different
XRONOS task. IMPORTANT NOTE: With caution this parameter
can be changed by the user.
EXAMPLES
1. Calculate statistical variables for an input time series with a
binning of 100 seconds and a start-stop=5000 seconds in one
interval (50 newbin per interval)
> lcstats cfile1="mydata.lc" window="-" dtnb=INDEF nbint=50
2. For an input time series (consisting of several files) with an
original binning of 400 seconds and a total length of 6 hours,
calculate statistical variables using a newbin of 800 seconds
over 2-hour intervals (9 newbin per interval for a total of 3
intervals).
> lcstats cfile1="@all.lis" window="-" dtnb=800 nbint=9
The parameter cfile1 is a file containing the list of filenames
for one time series (see "Filelist and Input File Options").
SEE ALSO
efold, efsearch, crosscor, autocor, powspec, lcurve, listdata,
timeskew xronwin, fits2qdp, ascii2lc.
BUGS
Report problems to angelini@lheavx.gsfc.nasa.gov and
xanprob@athena.gsfc.nasa.gov. Provide a detailed description of
the problem (with a log file if possible).
Please send reports of errors to : xanprob@athena.gsfc.nasa.gov HEASARC Home | Observatories | Archive | Calibration | Software | Tools | Students/Teachers/Public Last modified: Thursday, 06-May-2004 13:47:11 EDT HEASARC Staff Scientist Position - Applications are now being accepted for a Staff Scientist with significant experience and interest in the technical aspects of astrophysics research, to work in the High Energy Astrophysics Science Archive Research Center (HEASARC) at NASA Goddard Space Flight Center (GSFC) in Greenbelt, MD. Refer to the AAS Job register for full details. |
HEASARC Staff Scientist Position - Applications are now being accepted for a Staff Scientist with significant experience and interest in the technical aspects of astrophysics research, to work in the High Energy Astrophysics Science Archive Research Center (HEASARC) at NASA Goddard Space Flight Center (GSFC) in Greenbelt, MD. Refer to the AAS Job register for full details.
