ftmeld is a task that merges FITS two or more separate FITS tables. It is similar to ftmerge in that both of the tasks append (concatenate) the rows of the input tables, but ftmeld is much more flexible in handling situations where the input tables do not have exactly identical structures.
ftmeld is designed to be essentially a drop-in replacement for ftmerge, but with added flexibility.
If you know that your input tables have identical column structures, then you should use ftmerge. ftmerge will operate more efficiently and quickly. However, if you are attempting to merge dissimilar tables ftmeld may be most useful. Usually you wish to merge tabular files, (such as mission-specific "filter files") which have very similar but not exactly the same column structure. However, ftmeld is flexible enough to handle any column structure, although that may not be advisable.
As mentioned, ftmeld is more flexible than ftmerge. Here are the differences.
ftmeld utilizes the capabilities of the CFITSIO library to transfer data between different numerical types seamlessly. It handles all standard FITS data column types, including numbers, strings, bits, logicals, complex values, vectors and multiply-dimensioned fields, as well as variable-length vectors and variable-length strings. In cases where no input column matches the output, a null value is used.
ftmeld does still enforce some restrictions upon the inputs for sanity's sake.
When transferring data from the input to the output, the data type is not required to be the same between the two files. For example, if the input file has column TIME stored as a 32-bit floating point number and the output file has TIME stored as a 64-bit floating point number, the 32-bit values will be converted to 64-bit values during the transfer.
FITS as a standard allows more complicated numerical storage techniques. The keywords TSCALn and TZEROn allow simple integer storage types to be stored in the file, but have a scale and offset applied when reading or writing the values. Thus, a column store as integer may "appear" to be a floating point data type. ftmeld will attempt to transfer these values from input to output using the appropriate scalings.
However, if the transfer would result in loss of numerical precision or numerical overflow, inaccurate values may result. This will normally result in an error message for the file in question. If you wish to allow numerical degradation, set errdegrade=NO, and the task will silently proceed - with the understanding that numerical fidelity may be lost.
Since null values must be assigned for every column, we discuss this topic more here. For columns in the output file that do not exist in the input file, a null value is used. Because ftmeld is designed with flexibility in mind, it must handle the possibility of missing data and for that reason null values must be assigned. Here is now ftmeld assigns null values:
COLUMN=VALUE,COLUMN=VALUE,...
| Data Type | Method or Value Used |
|---|---|
| 'E' 32-bit floating point | NaN |
| 'D' 64-bit double precision | NaN |
| 'C' 32-bit floating complex | NaN |
| 'M' 64-bit double complex | NaN |
| 'L' logical | ASCII 0 |
| 'X' bits | no null value possible |
| 'B' 8-bit byte (no TZEROn) | 255 |
| 'B' 8-bit byte (any TZEROn) | 0 |
| 'I' 16-bit short (no TZEROn) | -32768 |
| 'I' 16-bit signed short (TZEROn=32768) | 32767 |
| 'J' 32-bit long (no TZEROn) | -2147483648 |
| 'J' 32-bit signed long (TZEROn=2147483648) | 2147483647 |
| 'K' 64-bit long long (no TZEROn) | -9223372036854775808 |
| 'K' 64-bit signed long long (TZEROn=9223372036854775808) | 9223372036854775807 |
| Integer storage with TSCALn/TZEROn scaling | Lowest integer value |
For values stored as integers, ftmeld will typically choose the lowest possible integer value as the null value, except when "unsigned" integer types are used, in which case it uses the highest possible integer value. Please note that the null values listed are the raw null value as stored on disk, before any TSCALn/TZEROn scalings are performed.
If you are confused by these heuristics, you are best off to use the 'nullvalues' parameter to explicitly designate null values for all columns of interest.
The initial file structure, including FITS keywords, is taken from the first listed tables in the 'infile' parameter. This includes all table-level metadata keyword columns as well as HISTORY and COMMENT keywords.
FITS tables typically have descriptors for every column, such as TUNITn, which provide additional data about the column. ftmeld will copy metadata for the column from the first listed file in the infile list in which the column appears.
Since the above scheme can create ambiguity in how a FITS file is structured, you may wish to enforce some more structure. The best way to do this is to use as the first input file a template file, which contains all desired columns, with desired keywords, in their desired order. The table itself can be empty (no rows), but the template will establish the data types, ordering and metdata keywords for the output file. In addition to chosing a first template file, users may wish to select colmode='FIRST' as described below.
Like ftmerge, ftmeld also has a 'lastkey' parameter. You may list keywords to be taken from the last listed input file and copied to the output file. This is most useful for time-related keywords (and as long as the input file list is sorted in time order) such as TSTOP and DATE-END.
The 'colmode' and 'columns' parameters determine which columns survive to the output file.
Use 'columns' to list which columns are desired to be included, or excluded. The inclusion list is a comma-separated list of column names that will definitely be copied from each input file and others will be excluded. The exclusion list is a comma-separated list of column names, preceded by a '-' character, each of which will not be copied. You may either specify an inclusion list or exclusion list, but not both.
Another selection method is the 'colmode' parameter. Here is how colmode changes the behavior of the task:
Because of the way the task works internally, the task only scans each input file once. Therefore, it cannot know if columns are to be added or removed until it reaches a file which forces that decision. This may have efficiency implications. For example, when ftmeld'ing a long list of large files with colmode='UNION', and the last file has many additional columns, then the task will need to insert each of these columns in-place, resulting in a large amount of disk I/O. The same reasoning would apply if many columns were to be deleted. For this reason, users may consider the technique of using a "template" file as the first file and colmode='FIRST'.
Beyond ftmerge and ftmeld, there are other similar table-combining tasks to consider. The tasks >ftpaste and ftjoin can combine two tables in a column-wise fashion, i.e. appending columns instead of rows.