8.2.2 Notes about the stream filetype driver

The stream driver can be used to efficiently read a FITS file from the stdin file stream or write a FITS to the stdout file stream. However, because these input and output streams must be accessed sequentially, the FITS file reading or writing application must also read and write the file sequentially, at least within the tolerances described below.

CFITSIO supports 2 different methods for accessing FITS files on the stdin and stdout streams. The original method, which is invoked by specifying a dash character, "-", as the name of the file when opening or creating it, works by storing a complete copy of the entire FITS file in memory. In this case, when reading from stdin, CFITSIO will copy the entire stream into memory before doing any processing of the file. Similarly, when writing to stdout, CFITSIO will create a copy of the entire FITS file in memory, before finally flushing it out to the stdout stream when the FITS file is closed. Buffering the entire FITS file in this way allows the application to randomly access any part of the FITS file, in any order, but it also requires that the user have sufficient available memory (or virtual memory) to store the entire file, which may not be possible in the case of very large files.

The newer stream filetype provides a more memory-efficient method of accessing FITS files on the stdin or stdout streams. Instead of storing a copy of the entire FITS file in memory, CFITSIO only uses a set of internal buffer which by default can store 40 FITS blocks, or about 100K bytes of the FITS file. The application program must process the FITS file sequentially from beginning to end, within this 100K buffer. Generally speaking the application program must conform to the following restrictions:

• The program must finish reading or writing the header keywords before reading or writing any data in the HDU.
• The HDU can contain at most about 1400 header keywords. This is the maximum that can fit in the nominal 40 FITS block buffer. In principle, this limit could be increased by recompiling CFITSIO with a larger buffer limit, which is set by the NIOBUF parameter in fitsio2.h.
• The program must read or write the data in a sequential manner from the beginning to the end of the HDU. Note that CFITSIO's internal 100K buffer allows a little latitude in meeting this requirement.
• The program cannot move back to a previous HDU in the FITS file.
• Reading or writing of variable length array columns in binary tables is not supported on streams, because this requires moving back and forth between the fixed-length portion of the binary table and the following heap area where the arrays are actually stored.
• Reading or writing of tile-compressed images is not supported on streams, because the images are internally stored using variable length arrays.