Tag: gzip

Compressing FITs files

In the days of consumer high capacity storage things look great but modern CMOS cameras output fairly large sized FITs but coupled with short exposure sub one evening can result in a large amount of data to process.

So various solutions are available – block compression and byte compression. Taking one FITs image (test.fit) from a OSC CMOS we can compare different compression methods.

Windows Compression

Using the built compression accessed via a file’s advanced options the file is reduced from 49.8MB down to a 43.5MB of disk usage.

Gzip compression

Using the Linux gzip compression utility (v1.9) we perform a byte-level compression in userspace.

# cksum test.fit
4076709869 52223040 test.fit
# gzip -9 test.fit
# ls -lashi test.fit.gz
524306 33M -rw-r--r--. 1 root root 33M Jul 15 14:34 test.fit.gz
# du -sh test.fit.gz
33M     test.fit.gz
# cksum test.fit.gz
1554386539 34046021 test.fit.gz
# gunzip test.fit.gz
# cksum test.fit
4076709869 52223040 test.fit

Gzipping the 50MB fits files results in 33MB of disk usage. Using fpack (v1.7.0) results in similar results and the file format (fit.fz). The fit.fz file is supported by the processing tool Pixinsight.

# fpack -g -q0 -v test.fit

Block Compression

Here we utilise the block-level compression feature of IBM Spectrum Scale Developer Edition.

# ls -lashi test.fit
524305 50M -rw-r--r--. 1 root root 50M Jul 15 14:34 test.fit
# du -sh test.fit
50M     test.fit
# cksum test.fit
4076709869 52223040 test.fit
# /usr/lpp/mmfs/bin/mmchattr --compression z test.fit
# ls -lashi test.fit
524305 36M -rw-r--r--. 1 root root 50M Jul 15 14:34 test.fit
# cksum test.fit
4076709869 52223040 test.fit
# /usr/lpp/mmfs/bin/mmchattr --compression no test.fit
# du -sh test.fit
36M     test.fit

The compression is transparent to the user application and user and they seen as the original file, modified files will need to be recompressed. As this is at the file system block-level the checksum is the same if the file is compressed or not.

Pixinsight XIF Compression

Pixinsight offers the XISF format as an alternative to the FITs format. Existing FITs file can be converted to XISF using the BatchFormatConversion script. To utilise compression you need to supply the appropriate output hints.

Pixinsight Batch Format Conversion

The process console output shows the resulting size of the converted FITs image being reduced to 49% of its original size.

Pixinsight Process Console

Conclusion

Pixinsight XIF compression gives the best compression but the issue is that it is currently not currently an accepted standard and other utilities (astropy) do not have the ability to read them. Pixinsight can also read fit.fz files

NINA can save camera files as FITS or XISF, the XISF method offers LZ4, LZ4HC and Zlib compression but no gzip/fpack option for FITS.

For now I will continue to compress FITS to fit.gz or fit.fz to give me the benefits of space savings whilst still allowing me to use the data across Windows and Linux utilities.