Column and Keyword Filtering Syntax

The column filtering specifier can be used to create or modify table columns and/or header keywords in the input HDU. The presence of a column filter causes CFITSIO to create a new virtual file (usually in memory) that contains the specified modifications. Any other HDUs in the input file are also copied to the virtual file. The actual physical input file is not modified.

The column filtering specifier must begin with the keyword 'col' and is enclosed in square brackets following the name of an input FITS file. The column filter can be used to perform the following types of operations. More than one operation may be specified by separating them with commas or semi-colons.

Select a subset of columns from the input table

The list of columns to be included in the virtual table should be separated by commas or semi-colons. For example,

    file.fits[events][col X, Y]

will create a virtual copy of the input events table that only contains the X and Y columns. The '*' and '$' wildcard characters may be used to match the names of more than one column to be selected. It is sometimes more convenient to list the columns to be excluded from the virtual table, as in:

    file.fits[events][col -DETX, -DETY]
 or
    file.fits[events][col !DETX; !DETY]

These examples will copy all but the DETX and DETY columns to the virtual table.

Compute the values for a column

Append a new column to the table or recompute the values in an existing column. For example,

    file.fits[events][col PI = PHA * 1.05, X, Y]

will modify the values in the PI column, (or will create a new PI column if it does not already exist) using the arithmetic expression 'PHA * 1.05' (where PHA is a column in the input table) to calculate the values for the column. The expression that is used to compute the columns or keyword values can be arbitrarily complex and may be a function of other header keyword values and other columns. The full syntax and available functions for the expression are described in the calc_express help file.

In the above example, the virtual file will contain only 3 columns: PI, X, and Y. The '*' wildcard character may be used when specifying the name of the columns to be copied to the virtual file, as in these examples:

    file.fits[events][col PI = PHA * 1.05, X, Y, *DET]
    file.fits[events][col PI = PHA * 1.05, *]

In the first example, the virtual table will contain the PI, X, and Y columns and any other columns whose name ends with the string 'DET'. In the second example, the virtual table will contain all the columns present in the input table, as well as the PI column.

When a new column is created, an appropriate data type will be chosen by default based on the form of the expression used to calculate the values. Alternatively, one may specify the desired column data type in parentheses, followed the column name, as in '[col PI(I) = PHA * 1.05]'. The datatype is specified using the same code letters that are allowed for the value of the FITS TFORMn keyword (e.g., 'I' for integer*2, 'J' for integer*4, 'E' for real*4, and 'D' for real*8, etc. for binary tables, and 'I8', F12.3', 'E20.12', etc. for ASCII tables). The full range of FITS data types is supported including character string, logical, bit, byte and complex data types. Vector columns are also supported in binary tables.

Create or modify a header keyword

Create a new header keyword or modify the value of an existing keyword by prefixing the keyword name with a pound sign '#'. The expression must evaluate to a constant scalar (i.e., cannot have a column name in the expression). The comment string for the keyword may be specified in parentheses immediately following the keyword name (instead of supplying a datatype as in the case of creating a new column). For example:

    file.fits[events][col #VEL(radial velocity) = 23.4]

will create the keyword

  VEL     =                   23.4 / radial velocity

COMMENT and HISTORY keywords may also be created with the following syntax:

   #COMMENT = 'This is a comment keyword'
   #HISTORY = 'This is a history keyword'

Note that the equal sign and the quote characters will be removed, so that the resulting header keywords in these cases will look like this:

   COMMENT This is a comment keyword
   HISTORY This is a history keyword

These two special keywords are always appended to the end of the header and will not affect any previously existing COMMENT or HISTORY keywords.

It is possible to delete an existing keyword using a preceding '-'. Either of these examples will delete the keyword named VEL.

  -VEL;
  -#VEL;

Wildcard Patterns to Delete Columns or Keywords

It is possible to use wildcard syntax to delete either keywords or columns that match a pattern. Recall that to delete either a keyword or a column, precede its name with a '-' character.

Wildcard patterns are: '*', which matches any string of characters; '?', which matches any single character; and '#' which matches any numerical string. For example these statements:

  -VEL*;       # remove single column (or keyword) beginning with VEL
  -VEL_?;      # remove single column (or keyword) VEL_? where ? is any character
  -#DEC_*;     # remove single keyword beginning with DEC_
  -#TUNIT#;    # remove single keyword TUNIT ending w. number

will remove the columns or keywords as noted. Be aware that if a '#' is not present, the CFITSIO engine will check for columns with the given name first, followed by keywords. The above expressions will only delete the first item which matches the pattern. If following columns or keywords in the same CHDU match the pattern, they will not be deleted. To delete zero or more keywords that match the pattern, add a trailing '+'.
-VEL*+; # remove all columns (or keywords) beginning with VEL -VEL_?+; # remove all columns (or keyword) VEL_? where ? is any character -#DEC_*+; # remove all keywords beginning with DEC_ -#TUNIT#+; # remove all keywords TUNIT ending w. number
Note that, as a 0-or-more matching pattern, this form will succeed if the requested column or keyword is not present. In that case, the deletion expression will silently proceed as if no deletion was requested.
Rename a column
Rename an existing column or keyword with the syntax 'NewName == OldName', as in:
file.fits[events][col NEWPHA == PHA]

Using a text file
For complex or commonly used operations, one can also place the operations into a text file and import it into the column filter using the syntax '[col @filename.txt]'. The operations can extend over multiple lines of the file, but multiple operations must still be separated by commas or semicolons. Any lines in the external text file that begin with 2 slash characters ('//') will be ignored and may be used to add comments into the file.
Multiple column filtering expressions

When using column filtering to open a file "on the fly," it is permitted to use multiple column filtering expressions. For example, the syntax
filename.fits[col *][col -Y][col Z=X+1]
would be treated as equivalent to joining the expressions with semicolons, or
filename.fits[col *; -Y;col Z=X+1]
Please note that if multiple column filtering expressions are used, it is not permitted to also use the
[col @filename.txt]
syntax in any of the individual expressions.
SEE ALSO
filenames, imfilter, rowfilter, binfilter, pixfilter, calc_express.