Use gzip fif files (.fif.gz?) about mne-python

[larsoner]

This would be pretty awesome. It would potentially save a lot of hard drive space.

[dengemann]

also +1 for this

On 07.10.2012, at 18:37, Eric89GXL [email protected] wrote:

This would be pretty awesome. It would potentially save a lot of hard drive space.

—
Reply to this email directly or view it on GitHub.

[mluessi]

I had the same idea a while ago. My hope was that using compression would also make reading and writing faster when using NFS volumes. I did an experiment , see

mluessi@386df6c

In my experiments it was very slow, maybe there is a way to make it faster :).

[agramfort]

looks like a cool sunday hack as I like to call this :)

how much slower ? do you know?

[larsoner]

Looks like bzip2 is an order of magnitude slower than gzip, so gzip might be the way to go even though it results in slightly larger file sizes, e.g.,:

http://tukaani.org/lzma/benchmarks.html

Although it might also make sense just to implement both while we're at it, and suggest that users use gzip for speed. Thoughts?

[larsoner]

Since that site used a stronger compression option for bzip (which is unfair), consider this site which didn't:

http://blog.terzza.com/linux-compression-comparison-gzip-vs-bzip2-vs-lzma-vs-zip-vs-compress/

Looks like compression is pretty similar, but decompression (where I'd imagine using it most for raw files and the like) is faster for gzip and zip...

[agramfort]

you should read this:

http://gael-varoquaux.info/blog/?p=159

[larsoner]

That is interesting. After reading it, I lean toward using gzip. It looks like zlib1 or zlib3 tended to do the best, and from what I understand those are abstractions of the same compression scheme used in the gzip format. The conclusions drawn from that site were:

...
4. Depending on the size of the data, it may be more efficient to store subsets in different files: it introduces ‘chunk’ that avoid filling in the memory too much (parameter cache_size in joblib’s code). In addition, data of a same nature tends to compress better.
5. The I/O stream or file object interfaces are abstractions that can hide the data movement and the creation of large temporaries. After experiments with GZipFile and StringIO/BytesIO I found it more efficient to fall back to passing around big buffer object, numpy arrays, or strings.
6. For reasons 4 and 5, I ended up avoiding the gzip module: raw access to the zlib with buffers gives more control. This explains a good part of the differences in read speed for pure arrays with numpy’s
...

However, for us, I imagine 4 won't be a big issue. When data are loaded with preload=True (which I imagine would satisfy most use cases), issue 5 shouldn't matter. We might have to see what happens without it...

In any case, the thing I like about gzip compared to some of the other options (using numpy's routines or compressing matrices individually and saving those) is that it maintains compatibility in the sense that, if I want, I can use a GUI to decompress the file and look at it. If we stored it in some other format, then a user wouldn't be able to do that. I would have no problem saving my data long-term in .gz format, but I'm not sure I'd be comfortable doing it with a format I'd have to go into python (or more likely, mne-python) to extract.

[agramfort]

I would investigate 2 approaches

1/ indeed .gz files that you can expand on any system
2/ or just gzip the data buffers and create a fif tag for compressed float
buffer matrices.

the issue with 2 is that mne and neuromag tools won't be able to read it.

I guess let's see if 1/ works (compress good and is fast enough on real
data)