Compressing FITs files

In the days of consumer high capacity storage things look great but modern CMOS cameras output fairly large sized FITs but coupled with short exposure sub one evening can result in a large amount of data to process.

So various solutions are available – block compression and byte compression. Taking one FITs image (test.fit) from a OSC CMOS we can compare different compression methods.

Windows Compression

Using the built compression accessed via a file’s advanced options the file is reduced from 49.8MB down to a 43.5MB of disk usage.

Gzip compression

Using the Linux gzip compression utility (v1.9) we perform a byte-level compression in userspace.

# cksum test.fit
4076709869 52223040 test.fit
# gzip -9 test.fit
# ls -lashi test.fit.gz
524306 33M -rw-r--r--. 1 root root 33M Jul 15 14:34 test.fit.gz
# du -sh test.fit.gz
33M     test.fit.gz
# cksum test.fit.gz
1554386539 34046021 test.fit.gz
# gunzip test.fit.gz
# cksum test.fit
4076709869 52223040 test.fit

Gzipping the 50MB fits files results in 33MB of disk usage. Using fpack (v1.7.0) results in similar results and the file format (fit.fz). The fit.fz file is supported by the processing tool Pixinsight.

# fpack -g -q0 -v test.fit

Block Compression

Here we utilise the block-level compression feature of IBM Spectrum Scale Developer Edition.

# ls -lashi test.fit
524305 50M -rw-r--r--. 1 root root 50M Jul 15 14:34 test.fit
# du -sh test.fit
50M     test.fit
# cksum test.fit
4076709869 52223040 test.fit
# /usr/lpp/mmfs/bin/mmchattr --compression z test.fit
# ls -lashi test.fit
524305 36M -rw-r--r--. 1 root root 50M Jul 15 14:34 test.fit
# cksum test.fit
4076709869 52223040 test.fit
# /usr/lpp/mmfs/bin/mmchattr --compression no test.fit
# du -sh test.fit
36M     test.fit

The compression is transparent to the user application and user and they seen as the original file, modified files will need to be recompressed. As this is at the file system block-level the checksum is the same if the file is compressed or not.

Pixinsight XIF Compression

Pixinsight offers the XISF format as an alternative to the FITs format. Existing FITs file can be converted to XISF using the BatchFormatConversion script. To utilise compression you need to supply the appropriate output hints.

Pixinsight Batch Format Conversion

The process console output shows the resulting size of the converted FITs image being reduced to 49% of its original size.

Pixinsight Process Console

Conclusion

Pixinsight XIF compression gives the best compression but the issue is that it is currently not currently an accepted standard and other utilities (astropy) do not have the ability to read them. Pixinsight can also read fit.fz files

NINA can save camera files as FITS or XISF, the XISF method offers LZ4, LZ4HC and Zlib compression but no gzip/fpack option for FITS.

For now I will continue to compress FITS to fit.gz or fit.fz to give me the benefits of space savings whilst still allowing me to use the data across Windows and Linux utilities.

Viewing Report Thursday 30th June 2022 Tenerife – Parador

21:30 – 06:30

Observing setup

First of all tonight I have taken darks for 300s that are needed for the earlier data. However I am still suffering from the QHY168c disconnecting from the laptop, this time in TSX. I have upgraded the software yesterday but this has not made a difference, so it must be something to do with the way there USB cable is connecting. I will attempt to troubleshoot when I get home. Right now I have switched to Ezcap again which seems to be ok.

I have setup over by Mark and will see what this location is like behind the pool building. I have found that the WiFi is weak or non existent. Also the staff room is nearby and the light is constantly on so maybe not the best place to image from.

@00:03 NGC 6352 30s -20℃ I took a single exposure of this.

@00:19 LDN 10 30s -20℃

LDN 10

@00:42 Haumea 30s -20℃

Haumea

@01:26 Makemake 30s -20℃

Makemake

@01:52 Pluto 30s -20℃

Pluto

@02:33 Eagle nebula 30s -20℃

M16 Eagle Nebula

@03:45 NGC 6520 &| B86 30s -20℃

NGC 6520 & B86 ‘Ink Spot’

Viewing Report Tuesday 28th June 2022 Tenerife – Parador

22:05 – 05:50

Pond at the Parador

@22:30 Omega Centauri 30s -20℃

Omega Centauri

@23:17 Centaurus A 30s -20℃

Centaurus A

@00:02 M24 30s -20℃

@01:33 M6 30s -20℃

@02:38 M25 30s -20℃

@03:26 LDN 564 300s -20℃

Flats taken

Darks 300s -20℃ (take tomorrow) / 0.35s -25℃ – DONE

Viewing Report Monday 27th June 2022 Tenerife – Parador

22:00 – 07:00

Skywatcher Esprit 120 ED

First thing tonight is to complete taking the darks from yesterday, so I need 20 x 120s -20℃, but also the flat darks I forgot for the 0.35ms flats. Then I will move to taking longer images tonight. The challenge with open clusters is that you need short exposures so you don’t oversaturate the stars.

@22:50 Omega Centauri 30s -20℃

Omega Centauri

@23:20 Centaurus A 300s -20℃

Centaurus A

@00:09 Makemake 120s -20℃ x 10

@00:29 Haumea 120s -20℃ x 10

@01:07 M8 Lagoon 300s -20℃ x 24 (focus drifted so review later images)

M8 Lagoon

@03:22 Comet C/2017 K2 PANSTARRS 120s -20℃ x 7

@03:53 M21 30s -20℃ x 90

@05:00 M20 & M21 120s x5

M20 & M21

@05:45 LDN 574 300s x 9 which was a mistake

LDN 574

Dark Flats – 0.35s -20℃ – DONE / 0.35s – 25℃ (tomorrow)

Flats – 0.35s -20℃ – DONE

Darks 30s -20℃ – DONE / 300 -20℃ (tomorrow) / 120 -20℃ DONE