Revisit compression

[tkittel]

We currently support compression or not simply by having gzip (external or internal) changing the files from .mcpl to .mcpl.gz and back. This was done prior to having provided mcpl_fat.c, in order to help mcstas/mcxtrace which did not link against zlib, and to avoid one extra dependency in general. It does provide some hassle for hook and script-writers, to figure out the final name of the output file.

However, given the existence of mcpl_fat.c it might not be so bad to simply assume zlib support everywhere. Thus, we could consider having internal compression and avoid dealing with .mcpl.gz at all.

If so, we might even consider compression in blocks of (say) 1000 particles at a time, or to otherwise use gzip blocks, in order to support more efficient backwards seeking or forward skips, which otherwise implies re-decompression of the entire amount of data before the target point.

The issue with that is of course that it makes particle lengths non-constant, introducing its own issues, as well as complicating the format.

Alternatively, we can investigate if perhaps a bit of custom compression (such as organising each block of 1024 particles in columns and using a custom run-length-encoding) would allow us to do away with zlib altogether. That would probably cost a bit of compression efficiency, but perhaps not too much.

This issue is clearly not for the near future.