Comments (6)
I can confirm that this works. Concatenated gz members are quite common in bioinformatics as well. The Bgzf standard uses this in combination with an index to allow random access on files or concurrent processing. The members are quite small (max 64Kb uncompressed data). Performance seems fine. Time to process a file is comparable to zcat
(when I use the zlib feature. Miniz is about 1.6 times slower, but this is also the case in single member gz files)
Partial example that uses the fastq reader from rust-bio
:
let mut reader = BufReader::new(file);
let mut r = fastq::Record::new();
loop {
//loop over all possible gzip members
match reader.fill_buf() {
Ok(b) => if b.is_empty() { break },
Err(e) => panic!(e)
}
//decode the next member
let gz = flate2::bufread::GzDecoder::new(&mut reader).unwrap();
let mut fqreader = fastq::Reader::new(gz);
//loop over all records in this member
loop {
match fqreader.read(&mut r) {
Ok(()) => {
if r.is_empty() {
//current gz member finished, more to decode?
break;
}
},
Err(err) => panic!(err)
}
//do stuff
}
}
from flate2-rs.
Unfortunately this has also been reported before (#35 and #23), but due to other divergences from zlib (see #40), I'm somewhat inclined to just implement a zlib-compatible gz stream as perhaps a separate type...
from flate2-rs.
Is there an easy workaround? I don't mind writing extra code on my end, but I don't know any other way to reliably check for EOF, since the GzDecoder takes ownership of its underlying reader and so I can't keep a borrowed reference to the File to check eof().
I suppose for my use-case I could get rid of flate entirely and pipe the output of gunzip -c
to stdin, but then I've got to wire everything up with shell scripts, and I'm limited to one input file.
from flate2-rs.
You may be able to do something like:
- Use
bufread::GzDecoder
instead ofread::GzDecoder
. - Use
&mut BufRead
instead of a by-valueT
- When one stream finishes decoding, check to see if the buffer has any more bytes in it, and if it does create a new
GzDecoder
to decode the next stream.
Although perhaps not tested, the GzDecoder
shouldn't consume any more bytes than it's supposed to, so if the streams are literally concatenated you should be able to detect this and just pick up where it left off.
I haven't tested this out yet, though, so it may not work :(
from flate2-rs.
To clarify, this is what I would expect:
extern crate flate2;
use std::io::prelude::*;
use std::io;
fn main() {
let mut v = Vec::new();
flate2::write::GzEncoder::new(&mut v, flate2::Compression::Best)
.write_all(b"foo")
.unwrap();
flate2::write::GzEncoder::new(&mut v, flate2::Compression::Best)
.write_all(b"bar")
.unwrap();
let mut data = &v[..];
io::copy(&mut flate2::bufread::GzDecoder::new(&mut data).unwrap(),
&mut io::stdout()).unwrap();
io::copy(&mut flate2::bufread::GzDecoder::new(&mut data).unwrap(),
&mut io::stdout()).unwrap();
}
It's crucial that you use bufread::GzDecoder
instead of read::GzDecoer
(as that may buffer too much), but otherwise while there's more data you can just keep decoding with a brand new GzDecoder
.
from flate2-rs.
Thanks for looking into this. I've been working on other parts of my system lately, but I'll implement this solution when I return to the Rust code.
from flate2-rs.
Related Issues (20)
- flate2::bufread::GzDecoder doesn't impl BufRead? HOT 3
- unsafe review: Potential (not actual) dangling pointers after inflate/deflate HOT 2
- total_in(&self) / total_out(&self) implementation for GzDecoder / GzEncoder / MultiGzDecoder HOT 2
- Issues with newly created file in read-write mode HOT 7
- Implement BufRead/Write for en/decoders alongside Read/Write
- rapidgzip
- Zlib succes while miniz_oxide fails HOT 5
- Testing validity of the data without the actual decompression
- Tree borrows violation occurs when using zlib backend HOT 5
- Some compressed files can only read a portion of the lines using GzDecoder. HOT 3
- question: Slowdown after upgrading from 1.0.26 to 1.0.28 HOT 8
- Using MultiGzDecoder for file with garbage after gzip data HOT 3
- Decoding a zip file returns the Error "corrupt deflate stream" HOT 2
- why GzDecoder can't read stream correct HOT 1
- Continue reading a stream after ZlibDecoder streams finishes HOT 8
- docs.rs failed to build flate2-1.0.29 HOT 1
- Add ability to set window_bits when using rust backend?
- unknown return code: -4 HOT 2
- libz-rs backend causes errors and panics HOT 2
- No git tag/release for 1.0.31 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flate2-rs.