HEASARC Staff Scientist Position - Applications are now being accepted for a Staff Scientist with significant experience and interest in the technical aspects of astrophysics research, to work in the High Energy Astrophysics Science Archive Research Center (HEASARC) at NASA Goddard Space Flight Center (GSFC) in Greenbelt, MD. Refer to the AAS Job register for full details.

Search:

[Advanced Search]



The HEASARC Database System

Transportable Database Aggregate Table (TDAT) Format

The TDAT file format is used to serialize HEASARC database tables with the data and metadata combined into a single, simple, structured file. TDAT files comprise a sequence of standard ASCII characters with the following generalized form: 
  <HEADER>
    ... keyword=value header lines
  <DATA>
    ... data lines, 1 line per record usually using a vertical bar ('|')
        character as a field delimiter
  <END>
Comments may be placed in the <HEADER> section. Comment lines are lines in which the first character(s) is(are) "#" or "//". Current software systems may allow arbitary text before the <HEADER> or after the <END> specifier, but only comments should be placed there.

TDAT Structure Keywords

<HEADER>
This required keyword designates the beginning of the header information. It should be the first non-comment token in the file and should be placed by itself on a line.
<DATA>
This required keyword flags the end of the header information and the beginning of the data section of the TDAT file. It should be placed by itself on a line.
<END>
This optional keyword specifies that the data section has ended and further processing of the file is not needed. Any text that comes after this keyword is ignored. It should be placed by itself on a line.
The above keywords are case-insensitive, but the uppercase versions are strongly preferred.

Header Format

All header information is in "name=value" format. The order of parameter definitions in the header is immaterial. Blank lines and comment lines (lines that start with "#" or "//") are ignored. Many of the parameter values (the portion of the parameter definition that comes after the equal sign) can be encased in quotes. Double quotes, single quotes, and back quotes are all recognized, but each pair of quotes must match in type. Whitespace on either side of the equal sign is ignored, as is any whitespace at the beginning of a line or trailing at the end of a line. Non-blank lines must be either comment lines or parameter definitions in the "name=value" format with exactly one such definition per line. Continuation lines are not supported.

Standard Header Keywords

Here is the list of recognized "name=value" parameters:
table_name = origin_tablename
Required. The supplied table name must either be a recognized system table ("zzgen", "zzext", "zzpar", and "zzrel") or begin with "origin_" where "origin" is the originator of the database. Currently, the only recognized origin is "heasarc"; however, that is easily changed. If an unrecognized origin is specified, the program will issue a warning but will continue with the ingest procedure anyway. Maximum length of a table name (including the "origin_" prefix) is 20 characters. Anything greater than that will be truncated and a warning issued. This value is stored in the zzgen.table_name field.
table_description = 'Here is the description.'
Optional, but strongly recommended. This is the description that goes into the metabase. This is stored in the zzgen.table_description field. Maximum description length is 80 characters. Anything greater than that will be truncated and a warning issued.
table_document_url = http://heasarc.gsfc.nasa.gov/W3Browse/observatoryname/tablename.html
Optional, but strongly recommended. This value is intended to be a URL for a document that describes the database table and its fields. This field is stored in the zzgen.table_document_url field.
table_security = private|public
Optional. The default is public. If table_security is public, then the software automatically grants select permissions to the public.
field[field_name] = type[:fmt][_unit] [[ucd]] [(index)|(key)] // description [// comment]
One entry required for each field in the table. Each line will be used to populate a row in the ZZPAR table. There are many (often optional) subfields in this line which are as follows:
field_name
This field is specified on the keyword side of the '='. It indicates the parameter name whose characteristics are being given in the line. the field name must be shorter than 24 characters.
type
The type gives the basic storage format for this field.
Recognized types are listed below. Unrecognized types are flagged as an error. The following are the supported field types. Recommended types are given in bold. 
	       int1     -- a one-byte integer
	       integer1 -- "    "        " (usage discouraged)
	       tinyint  -- "    "        " (usage discouraged)
	       int2     -- a two-byte integer
	       integer2 -- "    "        " (usage discouraged)
	       smallint -- "    "        " (usage discouraged)
	       int4     -- a four-byte integer
	       integer4 -- "     "        " (usage discouraged)
	       integer  -- "     "        " (usage discouraged)
	       float4   -- a four-byte floating point number
	       real     -- a synonym for 'float4' (usage discouraged)
	       float    -- an eight-byte floating point number (usage discouraged)
	       float8   -- "      "         "       "     "
	       char(#)  -- character string, # must be in the range 1 to 2000 (usage discouraged)
	       char#    -- character string, # must be in the range 1 to 2000
fmt
The fmt field is optional. It is signaled by a ':' after the type (i.e., without any separating whitespace). The fmt specifier is only allowed for floating-point and integer types. It allows one to specify the way floating-point values should be displayed. It uses the C or Perl printf() syntax, but without the leading percent symbol.
unit
The unit specifier is also optional and is signaled by a '_' after the type (or type:fmt). It is used to indicate the units that are used in the field. The unit is stored in the zzpar.parameter_unit field.
ucd
The Unified Content Descriptor (UCD) of the parameter. It is optional, but strongly recommended. The UCD is specified with square brackets around it, and it is case-sensitive. The UCD is stored in the zzpar.parameter_ucd field.
(index) or (key)
The index and key flags are optional but mutually exclusive. The "(index)" flag tells the program to create an index on this field after the data has been ingested into the database, while the "(key)" flag suggests that this field be used as the primary key for storing the database. If multiple key fields are specified then only the first one (alphabetically) may be used if the database system does not support multiple fields in the primary key index. If key is specified, the zzpar.parameter_is_index value is set to 'K'. If index is specified, it is set to 'Y'. Otherwise, it is set to 'N'.
description
The description is optional but strongly recommended. It gives the information that will be stored in the zzpar.parameter_description field and used to tell database users what this parameter means. It is truncated to a maximum of 80 characters.
comment
The comment field is optional and is used to attach some additional comment to the database. It is stored in the zzpar.parameter_comment field. Generally, it is used to provide additional information about field values. (E.g., "If type XP: pulse period of the pulsar"). It is truncated to a maximum of 80 characters.
The combination of type and format is stored in the zzpar.parameter_format field. This has a maximum of 24 characters and it is an error to exceed this.
Here are some example field definitions: 
field[ra]  = float8:.5f_degree  (key)   // Right Ascension
field[dec] = float8:.5f_degree  (key)   // Declination
field[flux]= float4:.2e_mJy     (index) // Flux Density Measured by Detector
field[time]= int4_mjd           (index) // Start Time of Observation
relate[field_name] = other_table_name(other_table_field_name) // description
Optional and obsolete. This definition specifies that a relationship exists between the supplied field_name and other_table_field_name of other_table_name. The optional description is strongly recommended. If either the table other_table_name or the parameter other_table_field_name does not exist, then a fatal error is issued, so make sure they exist first.
parameter_defaults = field_name1 field_name2 ...
Optional. This parameter definition designates the default column order for output in, for example, Browse. Not all field names have to be listed. This information is stored in the zzpar.parameter_defaults column. The first parameter listed is given the value 1, then next 2, and so forth. Parameters not specified in this list are given the value 0.
line[line_number] = field_name1 field_name2 ...
Required. This specifies the order of the fields on a given line_number in the data portion of the file. If a field name is specified here that was not also defined using a "field[field_name]=..." definition, then a fatal error is issued. The syntax allows the data to be given using multiple lines in the <DATA> section; however, it is recommended that only a single data line be used for each record input, i.e., that only line[1] be specified.
record_delimiter = "record_delimiter_string"
Optional and deprecated. Default is no record delimiter at all. The following list standard escaped character sequences and their equivalent meanings can be used: \t (tab), \b (backspace), \r (carriage return), \f (form feed), \v (vertical tab), \a (audible alert/bell), and \### (where ### is a number between 1 and 127 and represents the ASCII character with that numerical code). Note: Specifying a record delimiter value of "" is interpreted as a single blank line between records.
field_delimiter = "field_delimiter_character(s)"
Optional and deprecated. Default is the "|" character. The last field on the line should be delimited. Feature: You can specify more than just one character as field delimiters. For example, if the data file uses both "|" (vertical bar) and "!" (exclamation point) as delimiters in different places, you can set 'field_delimiter = "|!"' and both characters will be interpreted as field delimiters. The field_delimiter values can use the same escaped character sequences as used in record_delmiter values. For example, 'field_delimiter = "|\t"' would interpret both the vertical bar character and the tab character as valid field delimiters for the data. Note: Although you can specify a field delimiter other than the "|" character, "|" characters may not be used in any fashion other than as field delimiters.

Other Keywords

Any other "keyword=value" pairs other than those given above are treated as virtual parameters.
virtual_parameter_name = virtual_parameter_value
For example, 'observatory_name = ASCA'. The HEASARC has come up with a few local conventions for virtual parameters. For example, virtual parameter values that refer to database fields, the database field name is preceded by the "@" symbol. Case does not matter for virtual parameter names. All virtual parameter names are automatically converted to lowercase by the HDBingest software.
The case of most parameter definitions does not matter. Most are converted to lowercase internally. The only exceptions are the following: field descriptions, field comments, the table_description value, the table_document_url value, virtual parameter values (as described in the previous paragraph), the record_delimiter value, and the field_delimiter value.

Data Format

The data format is simply a sequence of lines which have record delimiters and field specified as above. Note that for character fields, leading spaces are significant. but are ignored for numeric fields. Since floating point fields are stored as ASCII values, it should be recognized that there may be small changes due to rounding error when these data are transferred among databases. Specifying the display format for floating point fields is therefore strongly recommended.

Null data fields for parameters are indicated by a zero-length field.

TDAT Example

An example TDAT file for the Messier catalog is given below. This example was generated by the HDBexgest program, but unless otherwise indicated other software may wish to provide header data in the same order. Only a few of the data records are included for the sake of brevity. 
<HEADER>
#
#         TABLE: heasarc_messier
#    TOTAL ROWS: 109
#
table_name = xx_messier
table_description = "Messier Nebulae Catalog"
table_document_url = http://heasarc.gsfc.nasa.gov/W3Browse/general-catalog/messier.html
table_security = public
#
# Table Parameters
#
field[alt_name] = char10  (index) // Alternate designation
field[bii] = float8_degree (index) // Galactic Latitude
field[class] = int2  (index) // Browse Object Classification
field[constell] = char4  (index) // Constellation of Origin
field[dec] = float8:.4f_degree (index) // Declination
field[dimension] = char6_arcmin (index) // Dimensions of the Source
field[lii] = float8_degree (index) // Galactic Longitude
field[name] = char6  (index) // Source designation
field[notes] = char50  (index) // Notes
field[object_type] = char2  (index) // Object Category
field[ra] = float8:.4f_degree (index) // Right Ascension
field[vmag] = float4:4.1f  (index) // Visual Magnitude
field[vmag_uncert] = char2  (index) // Magnitude Uncertainty
#
parameter_defaults = name alt_name ra dec constell dimension vmag vmag_uncert class
#
# Virtual Parameters
#
declination = @dec
default_search_radius = 60
equinox = 2000
frequency_regime = Optical
observatory_name = GENERAL CATALOG
right_ascension = @ra
table_priority = 3
table_type = Object
target_name = @name
unique_key = name
#
# Relationship Definitions
#
relate[class] = heasarc_class(class_id)
#
# Data Format Specification
#
line[1] = alt_name bii class constell dec dimension lii name notes object_type ra vmag vmag_uncert
#
<DATA>
NGC 6809|-23.2733634|3080|SGR|-30.9666947708543|19|8.7909942|M 55||GB|294.99980605110801|7.0||
NGC 6715|-14.0974428|3080|SGR|-30.483349176838999|9|5.6077231|M 54||GB|283.77480378105798|7.7||
NGC 6121|15.9648287|3080|SCO|-26.5333061473825|26|350.9692185|M 4||GB|245.89981204924899|5.9||
NGC 1904|-29.2898547|3080|LEP|-24.550010243430599|9|227.2840613|M 79||GB|81.124872363673802|8.0||
NGC 2447|.1495137|3600|PUP|-23.866637331244299|22|240.0587793|M 93||OC|116.149868339422|6.2|:|
NGC 7099|-46.8435102|3080|CAP|-23.1833879909756|11|27.1771312|M 30||GB|325.09982631146403|7.5||
NGC 6531|-0.43694830|3600|SGR|-22.500001387839401|13|7.7120967|M 21||OC|271.149814658205|5.9||
NGC 2287|-10.226592|3600|CMA|-20.733319763806499|38|231.0978149|M 41||OC|101.74986693951899|4.5||
IC4725|-4.4562027000|3600|SGR|-19.250009170361899|32|13.5636638|M 25||OC|277.899819297269|4.6||
NGC 6494|2.872862700|3600|SGR|-19.016665704498902|27|9.8346853|M 23||OC|269.199819080682|5.5||
...
<END>
Documentation prepared by the HEASARC Database Group
HEASARC Home | Observatories | Archive | Calibration | Software | Tools | Students/Teachers/Public
Last modified: Friday, 26-Jul-2024 16:28:36 EDT

HEASARC Staff Scientist Position - Applications are now being accepted for a Staff Scientist with significant experience and interest in the technical aspects of astrophysics research, to work in the High Energy Astrophysics Science Archive Research Center (HEASARC) at NASA Goddard Space Flight Center (GSFC) in Greenbelt, MD. Refer to the AAS Job register for full details.