Premature EOF on WARC files about flate2-rs

[veldsla]

I can confirm that this works. Concatenated gz members are quite common in bioinformatics as well. The Bgzf standard uses this in combination with an index to allow random access on files or concurrent processing. The members are quite small (max 64Kb uncompressed data). Performance seems fine. Time to process a file is comparable to zcat (when I use the zlib feature. Miniz is about 1.6 times slower, but this is also the case in single member gz files)

Partial example that uses the fastq reader from rust-bio:

let mut reader = BufReader::new(file);
let mut r = fastq::Record::new();

loop {
    //loop over all possible gzip members
    match reader.fill_buf() {
        Ok(b) => if b.is_empty() { break },
        Err(e) => panic!(e)
    }

    //decode the next member
    let gz = flate2::bufread::GzDecoder::new(&mut reader).unwrap();
    let mut fqreader = fastq::Reader::new(gz);

    //loop over all records in this member
    loop {
        match fqreader.read(&mut r) {
            Ok(()) => {
                if r.is_empty() {
                    //current gz member finished, more to decode?
                    break;
                }
            },
            Err(err) => panic!(err)
        }
        //do stuff
    }
}

[alexcrichton]

Unfortunately this has also been reported before (#35 and #23), but due to other divergences from zlib (see #40), I'm somewhat inclined to just implement a zlib-compatible gz stream as perhaps a separate type...

[nostrademons]

Is there an easy workaround? I don't mind writing extra code on my end, but I don't know any other way to reliably check for EOF, since the GzDecoder takes ownership of its underlying reader and so I can't keep a borrowed reference to the File to check eof().

I suppose for my use-case I could get rid of flate entirely and pipe the output of gunzip -c to stdin, but then I've got to wire everything up with shell scripts, and I'm limited to one input file.

[alexcrichton]

You may be able to do something like:

• Use bufread::GzDecoder instead of read::GzDecoder.
• Use &mut BufRead instead of a by-value T
• When one stream finishes decoding, check to see if the buffer has any more bytes in it, and if it does create a new GzDecoder to decode the next stream.

Although perhaps not tested, the GzDecoder shouldn't consume any more bytes than it's supposed to, so if the streams are literally concatenated you should be able to detect this and just pick up where it left off.

I haven't tested this out yet, though, so it may not work :(

[alexcrichton]

To clarify, this is what I would expect:

extern crate flate2;

use std::io::prelude::*;
use std::io;

fn main() {
    let mut v = Vec::new();
    flate2::write::GzEncoder::new(&mut v, flate2::Compression::Best)
                             .write_all(b"foo")
                             .unwrap();
    flate2::write::GzEncoder::new(&mut v, flate2::Compression::Best)
                             .write_all(b"bar")
                             .unwrap();

    let mut data = &v[..];

    io::copy(&mut flate2::bufread::GzDecoder::new(&mut data).unwrap(),
             &mut io::stdout()).unwrap();
    io::copy(&mut flate2::bufread::GzDecoder::new(&mut data).unwrap(),
             &mut io::stdout()).unwrap();
}

It's crucial that you use bufread::GzDecoder instead of read::GzDecoer (as that may buffer too much), but otherwise while there's more data you can just keep decoding with a brand new GzDecoder.

[nostrademons]

Thanks for looking into this. I've been working on other parts of my system lately, but I'll implement this solution when I return to the Rust code.