The HEASARC Database System

Ingest Software

Using HDBingest:

First, one needs to set a few environment variables.

INGEST_TMPDIR : Required. This needs to point to a directory where HDBingest can store some temporary files.

INGEST_BACKUPDIR : Required. This designates the directory where HDBingest will put the backup files for any table that it has to modify.

HEASARC_DATABASE : Optional. This is the database into which HDBingest will ingest the data. Default is "heasarc". This database *must* already have all the metabase tables properly set up.

HDB_PORT : Optional. This is the port number of the metabase server. Default is 40975 which is the metabase server for the "heasarc" database.

After constructing a TDAT file with correct header information and data, ingesting is as simple as the following command:

% HDBingest new_table.tdat

If new_table already exists, then the above would only *append* the rows in new_table.tdat to new_table. Changes in virtual parameters, parameter min/max values, and relations are updated, but field types, field names, and what fields are indexed are *not* updated. In order to update this information, you need to completely rebuild the table using the following:

% HDBingest -rebuild new_table.tdat

This drops the table and all associated indexes from the database and deletes all rows related to new_table in the metabase tables first. Then, HDBingest proceeds to ingest the information from new_table.tdat as if from scratch.

If the table already exists and all you want to do is update the relavant metabase entries for that table, then you can use the following:

% HDBingest -header tablename.hdr

Changes in virtual parameters, parameter min/max values, and relations are updated, but field types, field names, and what fields are indexed are *not* updated. If you want to change those, then you must rebuild the table instead. The file used to update the metabase information can be a regular TDAT file or a file consisting of just the header portion of a TDAT file.

Please note that HDBingest implements a lock-out mechanism so that only one person/process can ingest into the database at one time. It does this by temporarily creating a table in the database named "zztmplock" which contains the time the lock was acquired and the e-mail address of the person who acquired the lock. Should an error occur during the ingest process, HDBingest will attempt to restore the metabase to its previous status if any changes had been made to it and/or restore the previous contents of the table. Should either attempt at restoration fail, the lock will stay intact so that the user can verify the state of the metabase and/or table personally before allowing others to ingest into the database. The lock can then be removed by manually dropping the "zztmplock" table from the database.