frommi / miniz_oxide Goto Github PK
View Code? Open in Web Editor NEWRust replacement for miniz
License: MIT License
Rust replacement for miniz
License: MIT License
miniz_oxide
deep in dependency tree makes my project uncompilable on WASM with stable Rust.
I wanted to make a PR, but noticed that it's actually already fixed on master >2 months ago, but wasn't released to crates.io.
Can you please publish a new release?
The function seems to get confused when the vector is partially drained between calls. This contradicts the documentation which specifies it should never panic.
Execute the test render_images
at this exact commit of image-png
: https://github.com/image-rs/image-png/tree/cb9a340e6e1b9cd4dbcc1bb69368399d2354da77
I don't have more minimized test cases at the moment, sorry. It's getting attached as soon as one is found.
Note the manual streaming inflate implementation. It transfers 'fully decoded' data to a separate buffer provided by the caller, modifying the position of the constructed Cursor
. According to documentation it leaves 32KB data in the intermediate buffer. Tracing shows that the second call to decompress
panics already and skipping the call to transfer_finished_data
also avoids it.
The PNG
format is specifically designed for decoding a rough view of image in a small subset at the beginning of the image called pass extraction such as adam7 interlacing. The benefit is substantially reduced if the image data can only be decompressed as a whole.
It also puts the decoding party at risk of resource exhaustion since it may be impossible to progress without violating limits on buffer allocations.
Context: we would like to switch inflate
implementation in image-png
to reduce the amount of unsafe
code in dependencies, better stability and performance.
Ideally we should never panic of course, but there could still be bugs, and if there are, it shouldn't end up in a segfault.
See this
I've tried with removing no_mangle and extern_c from mz_inflate, so that doesn't seem to be the issue.
tinfl_decompress is implemented as a coroutine (state machine) and is a bit of a mess. tdefl is divided into functions so I was able to rewrite it function by function and check that nothing broke, but it won't work as straightforward for tinfl.
I propose to write a big match over enum of all the states and at first delegate them to C tinfl_decompress and rewrite them one by one in terms of match arms.
Now that the C API stuff is split off, we should prepare and publish the rust crate to crates.io. Possibly also the C API.
The internal compressor/decompressor structs do not match the layout of the C versions, so we want to generate headers with the correct layout rather than using the C headers for the original library. The C zip tests/functions segfault at the moment as the decompressor structure is put on the stack, and the init function in C is just a macro that zeroes a value.
CBindgen currently doesn't handle function pointers and boxes inside options (Whcih can be exported to c as pointers). Need to either be fixed upstream, or we will want to use raw pointers here which means a bit more unsafety.
To reproduce, run the following program (with miniz_oxide 0.3.6
):
fn main() {
miniz_oxide::deflate::compress_to_vec(&[], 6);
}
Expected result: it should terminate instantly.
Actual result: it does not terminate.
Same for
fn main() {
miniz_oxide::deflate::compress_to_vec(&[0], 6);
}
The pure rust implementation of deflate gives a somewhat unsatisfactory compression result, even using max compression level.
Somehow, the zlib binding from here is even worse, which is very strange since my benchmark is from Node.js's zlib, using zlib's default compression level.
Consider the following program - it shouldn't be too much of a stretch to wonder how this library isn't able to produce the same result as COMPRESSED
at default, let alone max.
use miniz_oxide::deflate::compress_to_vec_zlib;
use miniz_oxide::deflate::compress_to_vec;
use miniz_oxide::inflate::decompress_to_vec;
const TEXT: &'static str = concat![
"Chunk one\n",
"data data\n",
"\n",
"Chunk two\n",
"data data\n",
"\n",
"Chunk three\n",
"data data\n",
];
// Generated in Node.js using it's zlib bindings.
// Zlib default compression level.
//
// Taken from my defunct project:
// https://github.com/Fishrock123/zlib-transform/blob/master/test/test-basic.js
const COMPRESSED: &'static [u8] = &[
0x73, 0xce, 0x28, 0xcd, 0xcb, 0x56, 0xc8, 0xcf, 0x4b, 0xe5, 0x4a, 0x49, 0x2c, 0x49, 0x54, 0x00, 0x11, 0x5c, 0x5c, 0xce, 0x60, 0xc1, 0x92, 0xf2, 0x7c, 0x2c, 0x82, 0x19, 0x45, 0xa9, 0xc8, 0x6a, 0x01,
];
fn main() {
let compressed = compress_to_vec(TEXT.as_bytes(), 10);
println!("miniz_oxide : {:x?}", compressed);
let compressed = compress_to_vec_zlib(TEXT.as_bytes(), 10);
println!("zlib : {:x?}", compressed);
println!("node.js zlib: {:x?}", COMPRESSED);
let text = decompress_to_vec(COMPRESSED).unwrap();
assert_eq!(TEXT.as_bytes(), &text[..]);
assert_eq!(TEXT, std::str::from_utf8(&text).unwrap());
}
miniz_oxide : [6d, ca, b1, 9, 0, 30, 8, 5, d1, fe, 4f, e1, 2e, 4e, 22, 44, 10, 2, a, c1, 90, f5, 43, 52, 59, d8, 5c, f1, 38, b6, ed, 93, c2, 15, 43, 52, e8, 5, e0, 8f, 79, a2, 41, 5b, 5a, df, b]
zlib : [78, da, 6d, ca, b1, 9, 0, 30, 8, 5, d1, fe, 4f, e1, 2e, 4e, 22, 44, 10, 2, a, c1, 90, f5, 43, 52, 59, d8, 5c, f1, 38, b6, ed, 93, c2, 15, 43, 52, e8, 5, e0, 8f, 79, a2, 41, 5b, 5a, df, b, ba, 21, 15, 4c]
node.js zlib: [73, ce, 28, cd, cb, 56, c8, cf, 4b, e5, 4a, 49, 2c, 49, 54, 0, 11, 5c, 5c, ce, 60, c1, 92, f2, 7c, 2c, 82, 19, 45, a9, c8, 6a, 1]
(This was noticed when building https://github.com/Fishrock123/tide-compress.)
Hi!
I read through #26, and we could add a small hack to stub the left-over libc calls to also be able to compile miniz_oxide_c_api
to Wasm. This is all following what was done to use libsodium in alexcrichton/wasm-sodium (see the libc-stub folder).
The wasm32 branch in wehlutyk/miniz_oxide shows that, and it makes compiling flate2-rs
to wasm32 possible (see wehlutyk/flate2-rs/tree/wasm32). I use it to read compressed pdf files using lopdf
, all in the browser, and up to now it works like a charm.
Would you accept a PR for this? It would be great as we could then merge the changes for flate2-rs
and lopdf
and have them work on wasm32. But seeing as it's a bit of hack I thought you might want to wait till libc gets implementations for wasm32. Then again it's a useful shim in the meantime :)
At least decompression of raw blocks and compression doesn't work.
Tested using cross
E.g tests run fine on mips64el, but not mips64 deflate::test::compress_small
and inflate::tinfl_oxide::test::raw_block
are failing with incorrect output. Haven't tested the C API yet.
Working on fixing it.
An interesting problem was reported at https://gitter.im/rust-lang/rust?at=5a57cc7183152df26d5d5cfc.
Basically, miniz-sys
and miniz_oxide_c_api
expose the same C-API, so you can't link them both into the same process. So it might make sense to bypass C-API in flate2 and avoid polluting linker's namespace.
The generated fine does not include the referred compression level.
It should looks like:
Level | ZLIB | GZIP
1 | 78 01 | 1F 8B
2 | 78 5E | 1F 8B
3 | 78 5E | 1F 8B
4 | 78 5E | 1F 8B
5 | 78 5E | 1F 8B
6 | 78 9C | 1F 8B
7 | 78 DA | 1F 8B
8 | 78 DA | 1F 8B
9 | 78 DA | 1F 8B
However, the second byte is always 0x01
. This happens using the default
feature. Using zlib
backend it works as expected.
Originally reported at: rust-lang/flate2-rs#225
22: std::panic::catch_unwind
at /checkout/src/libstd/panic.rs:365
23: mz_inflate
at /home/sanmai/.cargo/bin/registry/src/github.com-1ecc6299db9ec823/miniz_oxide_c_api-0.1.1/src/lib.rs:104
24: flate2::mem::Decompress::decompress
at /home/sanmai/.cargo/bin/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.1/src/mem.rs:368
25: <flate2::mem::Decompress as flate2::zio::Ops>::run
Found as part of the Rust compiler build.
miniz version - miniz_oxide-0.3.5
Rust version tag : release v1.41.1
Channel : nightly
config.toml
[llvm]
ninja = true
link-jobs = 1
version-suffix = "-rust"
release-debuginfo = true
use-libcxx = true
use-linker = "lld"
[rust]
optimize = true
debug = true
debuginfo-level = 2
debuginfo-level-rustc = 2
debuginfo-level-std = 2
debuginfo-level-tools = 2
channel = "nightly"
[build]
extended = true
verbose = 1
during x.py install
Dist rustc stage2 (x86_64-unknown-linux-gnu)
running: "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/x86_64-unknown-linux-gnu/stage0-tools-bin/fabricate" "generate" "--product-name=Rust" "--rel-manifest-dir=rustlib" "--success-message=Rust-is-ready-to-roll." "--image-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-image" "--work-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist" "--output-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/dist" "--non-installed-overlay" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-overlay" "--package-name=rustc-nightly-x86_64-unknown-linux-gnu" "--component-name=rustc" "--legacy-manifest-dirs=rustlib,cargo"
thread '<unnamed>' panicked at 'attempt to add with overflow', /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/core.rs:1669:31
stack backtrace:
0: backtrace::backtrace::libunwind::trace
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
1: backtrace::backtrace::trace_unsynchronized
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
2: std::sys_common::backtrace::_print_fmt
at src/libstd/sys_common/backtrace.rs:77
3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
at src/libstd/sys_common/backtrace.rs:61
4: core::fmt::write
at src/libcore/fmt/mod.rs:1028
5: std::io::Write::write_fmt
at src/libstd/io/mod.rs:1412
6: std::sys_common::backtrace::_print
at src/libstd/sys_common/backtrace.rs:65
7: std::sys_common::backtrace::print
at src/libstd/sys_common/backtrace.rs:50
8: std::panicking::default_hook::{{closure}}
at src/libstd/panicking.rs:188
9: std::panicking::default_hook
at src/libstd/panicking.rs:205
10: std::panicking::rust_panic_with_hook
at src/libstd/panicking.rs:464
11: std::panicking::continue_panic_fmt
at src/libstd/panicking.rs:373
12: rust_begin_unwind
at src/libstd/panicking.rs:302
13: core::panicking::panic_fmt
at src/libcore/panicking.rs:139
14: core::panicking::panic
at src/libcore/panicking.rs:70
15: miniz_oxide::deflate::core::compress_inner
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/core.rs:0
16: miniz_oxide::deflate::core::compress
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/core.rs:2070
17: miniz_oxide::deflate::stream::deflate
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/stream.rs:52
18: <flate2::ffi::rust::Deflate as flate2::ffi::DeflateBackend>::compress
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/ffi/rust.rs:149
19: flate2::mem::Compress::compress
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/mem.rs:297
20: flate2::mem::Compress::compress_vec
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/mem.rs:322
21: <flate2::mem::Compress as flate2::zio::Ops>::run_vec
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/zio.rs:54
22: flate2::zio::Writer<W,D>::write_with_status
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/zio.rs:217
23: <flate2::zio::Writer<W,D> as std::io::Write>::write
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/zio.rs:256
24: <flate2::gz::write::GzEncoder<W> as std::io::Write>::write
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/gz/write.rs:149
25: std::io::Write::write_all
at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/io/mod.rs:1341
26: <installer::tarballer::RayonTee<A,B> as std::io::Write>::write_all::{{closure}}
at src/tools/rust-installer/src/tarballer.rs:175
27: rayon_core::join::join::call::{{closure}}
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/join/mod.rs:103
28: rayon_core::join::join_context::call_b::{{closure}}
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/join/mod.rs:130
29: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute::call::{{closure}}
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:113
30: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:317
31: std::panicking::try::do_call
at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:287
32: __rust_maybe_catch_panic
at src/libpanic_unwind/lib.rs:78
33: std::panicking::try
at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:265
34: std::panic::catch_unwind
at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:396
35: rayon_core::unwind::halt_unwinding
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/unwind.rs:17
36: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:119
37: rayon_core::job::JobRef::execute
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:59
38: rayon_core::registry::WorkerThread::execute
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:735
39: rayon_core::registry::WorkerThread::wait_until_cold
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:719
40: rayon_core::registry::WorkerThread::wait_until
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:693
41: rayon_core::registry::main_loop
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:813
42: rayon_core::registry::ThreadBuilder::run
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:58
43: <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{{closure}}
at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:103
command did not execute successfully: "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/x86_64-unknown-linux-gnu/stage0-tools-bin/fabricate" "generate" "--product-name=Rust" "--rel-manifest-dir=rustlib" "--success-message=Rust-is-ready-to-roll." "--image-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-image" "--work-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist" "--output-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/dist" "--non-installed-overlay" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-overlay" "--package-name=rustc-nightly-x86_64-unknown-linux-gnu" "--component-name=rustc" "--legacy-manifest-dirs=rustlib,cargo"
expected success, got: signal: 4
solana-labs/solana#5411 looks similar.
Cargo built with miniz_oxide
panics while unpacking clfft v0.3.1
package from crates.io. Found this bug using brand new tester.
Archives of miniz_oxide (and probably others) on crates.io do not include LICENSE which is against MIT license terms. Please include it.
zlib-ng has a bunch of tests to check for among other things previous zlib vulnerabilities, and to provide good coverage of the library.
It would be useful to do port these tests to check that miniz_oxide won't panic on these specific inputs.
As they are zlib-licensed we may want to put the tests in separate files if possible and maybe exclude them from the crate upload, not sure how zlib and mit licenses interact (though the tests won't be a part of the final library obviously).
It'd be nice to not include any bindings to zlib when unnecessary.
Exposing mz_crc32_oxide()
could be useful for writing a PNG encoder/decoder that uses miniz_oxide for zlib compression anyway.
I can see that it is part of miniz_oxide/src/lib.rs, but not of miniz_oxide/miniz_oxide/src/lib.rs. But I don't understand what the difference is, only that I could not access mz_crc32_oxide().
On a related note, how can you access the streaming decompression that is discussed in issue 24?
When opening small-ish images with png
crate more than 50% of time is spent in inflate::core::init_tree
function. This makes png
crate 3x slower than spng
according to spng crate benchmarks. Using larger images closes the gap.
Sample image demonstrating the issue (25Kb): https://github.com/aloucks/spng-rs/blob/master/spng/tests/test-002.png
Guarding against decompression bombs should be easy.
I've found and minified a file which causes the same symptoms as #13, but I don't think it's the same problem.
The file is pretty horribly invalid. Horribly, horribly invalid. Confuses the other tool I'm working on, too.
Test and data file:
FauxFaux@0f3b480
thread 'inf_issue_13' panicked at 'attempt to subtract with overflow', miniz_oxide/src/inflate/core.rs:958:25
...
10: miniz_oxide::inflate::core::decompress_fast
at miniz_oxide/src/inflate/core.rs:958
11: miniz_oxide::inflate::core::decompress_inner
at miniz_oxide/src/inflate/core.rs:1347
12: miniz_oxide::inflate::core::decompress
at miniz_oxide/src/inflate/core.rs:1003
13: miniz_oxide::inflate::decompress_to_vec_inner
at miniz_oxide/src/inflate/mod.rs:93
14: miniz_oxide::inflate::decompress_to_vec
at ./src/inflate/mod.rs:61
15: test::inf_issue_13
at miniz_oxide/tests/test.rs:20
There's still some unsafe usage in the C API glue that could be avoided, such as raw memory allocation.
Eventually we want to fully fix #16, but the innter C glue code may or may not be used by that.
As there is a little bit of endinaness and word-size dependent code, there should be CI tests for this to avoid any breakage.
There is a place that probably will break on not x86 arch. As well as performance issues with read/write_unaligned.
Hi, I'm developing a streaming .ZIP library that uses miniz_oxide for decompressing DEFLATE. Streaming is hard because you get the data in smaller chunks and want to decompress incrementally. Because of some peculiarities of the .ZIP format, you don't sometimes even know the size of the data beforehands. Here's my library (WIP): https://github.com/golddranks/stream_zipper
I've managed to get decompression to work when parsing files as one chunk. However, incremental decompression is giving me trouble. I'd like to ask some questions which I'd hope get eventually clarified in documentation:
What's the difference between FailedCannotMakeProgress
and NeedsMoreInput
states? If I hand miniz_oxide
a partial buffer (try git cloning
my repo and running cargo test test_partial_decompression -- --nocapture
and see the debug printing; it gives 400 bytes instead of the full buffer), I get FailedCannotMakeProgress
.
If I want to continue incrementally, what parameters am I supposed to pass? At the moment, I pass the rest of the input buffer (bytes after 400 EDIT: After making sure that it has indeed consumed 400 bytes up to that point!) and make a fresh output buffer, plus reuse the same DecompressorOxide
struct, but it returns a Failure
. If I pass the input buffer in one go, it manages to decompress the thing. (Edit: Is it already in a failed state after FailedCannotMakeProgress
so that's why it returns Failure
?)
If I understand right, DEFLATE stream format has a marker when the stream ends, so knowing the size beforehands isn't required. Is this correct?
I've been trying to compile a simple script that uses miniz_ozide and am running into a few errors when running cargo build
. They mostly look like they're related to the use of unstable library features in deflate. Output below, with the path edited to $HOME
.
Compiling miniz_oxide v0.3.2
Compiling rand_chacha v0.1.1
Compiling rand_pcg v0.1.2
Compiling rand v0.6.5
Compiling rustc_version v0.2.3
Compiling proc-macro2 v0.4.30
Compiling rand_hc v0.1.0
error[E0658]: use of unstable library feature 'try_from' (see issue #33417)
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:3:5
|
3 | use std::convert::TryInto;
| ^^^^^^^^^^^^^^^^^^^^^
error[E0658]: use of unstable library feature 'try_from' (see issue #33417)
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:6:5
|
6 | use std::convert::TryInto;
| ^^^^^^^^^^^^^^^^^^^^^
error[E0658]: use of unstable library feature 'int_to_from_bytes' (see issue #52963)
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:634:25
|
634 | let bytes = u64::to_le_bytes(self.bit_buffer);
| ^^^^^^^^^^^^^^^^
error[E0658]: use of unstable library feature 'try_from' (see issue #33417)
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:1124:52
|
1124 | let bytes: [u8; 4] = self.b.dict[pos..end].try_into().unwrap();
| ^^^^^^^^
error[E0658]: use of unstable library feature 'int_to_from_bytes' (see issue #52963)
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:1125:9
|
1125 | u32::from_le_bytes(bytes)
| ^^^^^^^^^^^^^^^^^^
error[E0277]: the trait bound `[u8; 4]: std::convert::From<&[u8]>` is not satisfied
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:1124:52
|
1124 | let bytes: [u8; 4] = self.b.dict[pos..end].try_into().unwrap();
| ^^^^^^^^ the trait `std::convert::From<&[u8]>` is not implemented for `[u8; 4]`
|
= note: required because of the requirements on the impl of `std::convert::TryFrom<&[u8]>` for `[u8; 4]`
= note: required because of the requirements on the impl of `std::convert::TryInto<[u8; 4]>` for `&[u8]`
error[E0658]: use of unstable library feature 'try_from' (see issue #33417)
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:1133:56
|
1133 | let bytes: [u8; 8] = self.b.dict[pos..pos + 8].try_into().unwrap();
| ^^^^^^^^
error[E0658]: use of unstable library feature 'int_to_from_bytes' (see issue #52963)
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:1134:9
|
1134 | u64::from_le_bytes(bytes)
| ^^^^^^^^^^^^^^^^^^
error[E0277]: the trait bound `[u8; 8]: std::convert::From<&[u8]>` is not satisfied
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:1133:56
|
1133 | let bytes: [u8; 8] = self.b.dict[pos..pos + 8].try_into().unwrap();
| ^^^^^^^^ the trait `std::convert::From<&[u8]>` is not implemented for `[u8; 8]`
|
= note: required because of the requirements on the impl of `std::convert::TryFrom<&[u8]>` for `[u8; 8]`
= note: required because of the requirements on the impl of `std::convert::TryInto<[u8; 8]>` for `&[u8]`
error[E0658]: use of unstable library feature 'try_from' (see issue #33417)
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:342:44
|
342 | let two_bytes = iter.as_ref()[..2].try_into().unwrap();
| ^^^^^^^^
error[E0658]: use of unstable library feature 'int_to_from_bytes' (see issue #52963)
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:343:9
|
343 | u16::from_le_bytes(two_bytes)
| ^^^^^^^^^^^^^^^^^^
error[E0277]: the trait bound `[u8; 2]: std::convert::From<&[u8]>` is not satisfied
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:342:44
|
342 | let two_bytes = iter.as_ref()[..2].try_into().unwrap();
| ^^^^^^^^ the trait `std::convert::From<&[u8]>` is not implemented for `[u8; 2]`
|
= note: required because of the requirements on the impl of `std::convert::TryFrom<&[u8]>` for `[u8; 2]`
= note: required because of the requirements on the impl of `std::convert::TryInto<[u8; 2]>` for `&[u8]`
error[E0658]: use of unstable library feature 'try_from' (see issue #33417)
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:357:54
|
357 | let four_bytes: [u8; 4] = iter.as_ref()[..4].try_into().unwrap();
| ^^^^^^^^
error[E0658]: use of unstable library feature 'int_to_from_bytes' (see issue #52963)
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:358:9
|
358 | u32::from_le_bytes(four_bytes)
| ^^^^^^^^^^^^^^^^^^
error[E0277]: the trait bound `[u8; 4]: std::convert::From<&[u8]>` is not satisfied
--> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:357:54
|
357 | let four_bytes: [u8; 4] = iter.as_ref()[..4].try_into().unwrap();
| ^^^^^^^^ the trait `std::convert::From<&[u8]>` is not implemented for `[u8; 4]`
|
= note: required because of the requirements on the impl of `std::convert::TryFrom<&[u8]>` for `[u8; 4]`
= note: required because of the requirements on the impl of `std::convert::TryInto<[u8; 4]>` for `&[u8]`
Compiling rand_isaac v0.1.1
error: aborting due to 15 previous errors
Some errors occurred: E0277, E0658.
For more information about an error, try `rustc --explain E0277`.
error: Could not compile `miniz_oxide`.
warning: build failed, waiting for other jobs to finish...
error: build failed
Hi,
Currently, miniz_oxide
is not buildable with wasm32-unknown-unknown
as target.
I would like to help to remove libc
to be able to use flate2
inside a browser.
Unfortunately I have no knowledge in miniz/zlib api.
Do you think it's doable? Can you give me guidance to do it?
From what I understand it's highly related to #3 .
Thanks.
See #68
@oyvindln, can you please give a link for your performance tests?
On Windows, the following code produces a stack overflow:
extern crate miniz_oxide;
use miniz_oxide::deflate::core::CompressorOxide;
struct Foo {
bar: CompressorOxide,
}
fn main() {
let foo = Foo { bar: CompressorOxide::new(0), };
}
Specifically, I get this error: thread 'main' has overflowed its stack
.
This does not happen in release mode, however.
I think the structure LZOxide
may be too large, as the overflow happens in it's new()
funtion.
Perhaps it is a good idea to allocate this data on the heap?
The README mentions that exporting miniz-compatible headers is blocked on an upstream cbindgen issue. However, a workaround for that issue was devised: mozilla/cbindgen#326 (comment)
So it should be possible to export layout-compatible structures once again.
The fuzz target currently invokes miniz_oxide
like this:
It does not seem to disable alder32 checks. Because of checksum verification fuzzers cannot reach the actually interesting decoding code.
Rust fuzzer integration provides special configuration option fuzzing
which is set when a binary is built with fuzzing instrumentation; you can use it to disable checksums via conditional compilation (example from lodepng-rust). Alternatively, you can adapt your fuzzing target to set the required configuration parameters to disable checksum verification (inflate crate provides a *_no_checksum function).
cc @oyvindln who has authored the fuzz target.
I can also provide you with an initial fuzzing corpus to kickstart fuzzing, both with and without zlib headers, obtained by fuzzing inflate
crate with afl-fuzz, libfuzzer and honggfuzz. If you're interested, let me know and I'll attach it here.
honggfuzz-rs provides further documentation, including enabling sanitizers to catch bugs in code under unsafe
. Or ping me or https://github.com/rust-fuzz if you have any questions regarding fuzzing.
Running `/var/tmp/portage/dev-rust/miniz_oxide-0.3.5/work/debug/deps/test-d07cae2084eaa2a5`
running 7 tests
test decompress_zero_code_len_2 ... ok
test decompress_zero_code_len_oom ... ok
test inf_issue_14 ... ok
test inf_issue_19 ... ok
test roundtrip_lvl_0 ... FAILED
test roundtrip_lvl_9 ... FAILED
test roundtrip_lvl_1 ... FAILED
failures:
---- roundtrip_lvl_0 stdout ----
thread 'roundtrip_lvl_0' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1084:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
---- roundtrip_lvl_9 stdout ----
thread 'roundtrip_lvl_9' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1084:5
---- roundtrip_lvl_1 stdout ----
thread 'roundtrip_lvl_1' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1084:5
failures:
roundtrip_lvl_0
roundtrip_lvl_1
roundtrip_lvl_9
test result: FAILED. 4 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out
RUST_BACKTRACE=full
--- roundtrip_lvl_0 stdout ----
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1165:5
stack backtrace:
0: 0x55c5d8263be4 - backtrace::backtrace::libunwind::trace::h93069dd45e563114
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
1: 0x55c5d8263be4 - backtrace::backtrace::trace_unsynchronized::ha81d63fb01c61f43
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
2: 0x55c5d8263be4 - std::sys_common::backtrace::_print_fmt::h5800670bd4856d97
at src/libstd/sys_common/backtrace.rs:77
3: 0x55c5d8263be4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8e952e7a864dce8d
at src/libstd/sys_common/backtrace.rs:61
4: 0x55c5d8286c5c - core::fmt::write::ha6cd837d898e4a61
at src/libcore/fmt/mod.rs:1030
5: 0x55c5d81e06d5 - std::io::Write::write_fmt::haa9265683e464bfb
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/io/mod.rs:1412
6: 0x55c5d825ea91 - std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt::he7c7a5301251df0f
at src/libstd/io/impls.rs:141
7: 0x55c5d8265e9e - std::sys_common::backtrace::_print::he810963362c39756
at src/libstd/sys_common/backtrace.rs:65
8: 0x55c5d8265e9e - std::sys_common::backtrace::print::hde04e6d280846a89
at src/libstd/sys_common/backtrace.rs:50
9: 0x55c5d8265e9e - std::panicking::default_hook::{{closure}}::h9d7792d5f2173a87
at src/libstd/panicking.rs:188
10: 0x55c5d8265b38 - std::panicking::default_hook::h9970cb21e5e5b2d4
at src/libstd/panicking.rs:202
11: 0x55c5d826659b - std::panicking::rust_panic_with_hook::hcbf43bd831dd8ef3
at src/libstd/panicking.rs:464
12: 0x55c5d826613e - std::panicking::continue_panic_fmt::h47b6e25372f76c95
at src/libstd/panicking.rs:373
13: 0x55c5d8266026 - rust_begin_unwind
at src/libstd/panicking.rs:302
14: 0x55c5d8283a7e - core::panicking::panic_fmt::h24ff570098569537
at src/libcore/panicking.rs:141
15: 0x55c5d8283b77 - core::result::unwrap_failed::hf7d72924a9a2d22c
at src/libcore/result.rs:1165
16: 0x55c5d81dd300 - core::result::Result<T,E>::unwrap::h0830fd1b36ee5052
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libcore/result.rs:933
17: 0x55c5d81ddcbc - test::get_test_file_data::h65896587825f1fc9
at tests/test.rs:11
18: 0x55c5d81dde04 - test::get_test_data::h65a752d85858b18c
at tests/test.rs:53
19: 0x55c5d81dde5c - test::roundtrip::h91d60028ff561479
at tests/test.rs:57
20: 0x55c5d81de698 - test::roundtrip_lvl_0::h2a97638a87236059
at tests/test.rs:81
21: 0x55c5d81df0da - test::roundtrip_lvl_0::{{closure}}::he8364aa236ed0c19
at tests/test.rs:80
22: 0x55c5d81deade - core::ops::function::FnOnce::call_once::h731f141a0b0e1982
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libcore/ops/function.rs:227
23: 0x55c5d81eb32f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h19a09fd199a0ff21
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/liballoc/boxed.rs:942
24: 0x55c5d826d1aa - __rust_maybe_catch_panic
at src/libpanic_unwind/lib.rs:79
25: 0x55c5d820690a - std::panicking::try::h1d1a1c5142fbfdd7
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/panicking.rs:265
26: 0x55c5d820690a - std::panic::catch_unwind::h200ca6210a82d128
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/panic.rs:396
27: 0x55c5d820690a - test::run_test_in_process::h766e5938de77f4da
at src/libtest/lib.rs:570
28: 0x55c5d820690a - test::run_test::run_test_inner::{{closure}}::h3d3ac74133b541d4
at src/libtest/lib.rs:473
29: 0x55c5d8205fe8 - test::run_test::run_test_inner::he77bc794d3ace3bd
at src/libtest/lib.rs:494
30: 0x55c5d8204569 - test::run_test::hba77973327a2fc7f
at src/libtest/lib.rs:530
31: 0x55c5d81fa954 - test::run_tests::h5a1270ddd5e0a473
at src/libtest/lib.rs:299
32: 0x55c5d81fa954 - test::console::run_tests_console::h29d755f301130a08
at src/libtest/console.rs:295
33: 0x55c5d82009e4 - test::test_main::hf2bc958b6a636d99
at src/libtest/lib.rs:120
34: 0x55c5d8201961 - test::test_main_static::h90326c29e77e672b
at src/libtest/lib.rs:139
35: 0x55c5d81de6b6 - test::main::h314b86d7399b1960
36: 0x55c5d81ddc00 - std::rt::lang_start::{{closure}}::hbaebdb534a364c8b
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/rt.rs:61
37: 0x55c5d8265fc3 - std::rt::lang_start_internal::{{closure}}::h040fbb64fb819fe8
at src/libstd/rt.rs:48
38: 0x55c5d8265fc3 - std::panicking::try::do_call::hfb8f62e9781fa17e
at src/libstd/panicking.rs:287
39: 0x55c5d826d1aa - __rust_maybe_catch_panic
at src/libpanic_unwind/lib.rs:79
40: 0x55c5d8266a7d - std::panicking::try::h2aa48346968c7185
at src/libstd/panicking.rs:265
41: 0x55c5d8266a7d - std::panic::catch_unwind::h8da95d989d5c8997
at src/libstd/panic.rs:396
42: 0x55c5d8266a7d - std::rt::lang_start_internal::h298ac725745c4eb6
at src/libstd/rt.rs:47
43: 0x55c5d81ddbd9 - std::rt::lang_start::hce40aa1f7d621390
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/rt.rs:61
44: 0x55c5d81de6ea - main
45: 0x7f5e423d1f2b - __libc_start_main
at ../csu/libc-start.c:308
46: 0x55c5d81dc18a - _start
47: 0x0 - <unknown>
---- roundtrip_lvl_1 stdout ----
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1165:5
stack backtrace:
0: 0x55c5d8263be4 - backtrace::backtrace::libunwind::trace::h93069dd45e563114
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
1: 0x55c5d8263be4 - backtrace::backtrace::trace_unsynchronized::ha81d63fb01c61f43
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
2: 0x55c5d8263be4 - std::sys_common::backtrace::_print_fmt::h5800670bd4856d97
at src/libstd/sys_common/backtrace.rs:77
3: 0x55c5d8263be4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8e952e7a864dce8d
at src/libstd/sys_common/backtrace.rs:61
4: 0x55c5d8286c5c - core::fmt::write::ha6cd837d898e4a61
at src/libcore/fmt/mod.rs:1030
5: 0x55c5d81e06d5 - std::io::Write::write_fmt::haa9265683e464bfb
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/io/mod.rs:1412
6: 0x55c5d825ea91 - std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt::he7c7a5301251df0f
at src/libstd/io/impls.rs:141
7: 0x55c5d8265e9e - std::sys_common::backtrace::_print::he810963362c39756
at src/libstd/sys_common/backtrace.rs:65
8: 0x55c5d8265e9e - std::sys_common::backtrace::print::hde04e6d280846a89
at src/libstd/sys_common/backtrace.rs:50
9: 0x55c5d8265e9e - std::panicking::default_hook::{{closure}}::h9d7792d5f2173a87
at src/libstd/panicking.rs:188
10: 0x55c5d8265b38 - std::panicking::default_hook::h9970cb21e5e5b2d4
at src/libstd/panicking.rs:202
11: 0x55c5d826659b - std::panicking::rust_panic_with_hook::hcbf43bd831dd8ef3
at src/libstd/panicking.rs:464
12: 0x55c5d826613e - std::panicking::continue_panic_fmt::h47b6e25372f76c95
at src/libstd/panicking.rs:373
13: 0x55c5d8266026 - rust_begin_unwind
at src/libstd/panicking.rs:302
14: 0x55c5d8283a7e - core::panicking::panic_fmt::h24ff570098569537
at src/libcore/panicking.rs:141
15: 0x55c5d8283b77 - core::result::unwrap_failed::hf7d72924a9a2d22c
at src/libcore/result.rs:1165
16: 0x55c5d81dd300 - core::result::Result<T,E>::unwrap::h0830fd1b36ee5052
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libcore/result.rs:933
17: 0x55c5d81ddcbc - test::get_test_file_data::h65896587825f1fc9
at tests/test.rs:11
18: 0x55c5d81dde04 - test::get_test_data::h65a752d85858b18c
at tests/test.rs:53
19: 0x55c5d81dde5c - test::roundtrip::h91d60028ff561479
at tests/test.rs:57
20: 0x55c5d81de68b - test::roundtrip_lvl_1::h7254383e6a5eedb4
at tests/test.rs:76
21: 0x55c5d81df0ba - test::roundtrip_lvl_1::{{closure}}::h1ea577cb5ee3e6cd
at tests/test.rs:75
22: 0x55c5d81dec0e - core::ops::function::FnOnce::call_once::hc9e28689b5a1b24f
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libcore/ops/function.rs:227
23: 0x55c5d81eb32f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h19a09fd199a0ff21
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/liballoc/boxed.rs:942
24: 0x55c5d826d1aa - __rust_maybe_catch_panic
at src/libpanic_unwind/lib.rs:79
25: 0x55c5d820690a - std::panicking::try::h1d1a1c5142fbfdd7
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/panicking.rs:265
26: 0x55c5d820690a - std::panic::catch_unwind::h200ca6210a82d128
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/panic.rs:396
27: 0x55c5d820690a - test::run_test_in_process::h766e5938de77f4da
at src/libtest/lib.rs:570
28: 0x55c5d820690a - test::run_test::run_test_inner::{{closure}}::h3d3ac74133b541d4
at src/libtest/lib.rs:473
29: 0x55c5d8205fe8 - test::run_test::run_test_inner::he77bc794d3ace3bd
at src/libtest/lib.rs:494
30: 0x55c5d8204569 - test::run_test::hba77973327a2fc7f
at src/libtest/lib.rs:530
31: 0x55c5d81fa954 - test::run_tests::h5a1270ddd5e0a473
at src/libtest/lib.rs:299
32: 0x55c5d81fa954 - test::console::run_tests_console::h29d755f301130a08
at src/libtest/console.rs:295
33: 0x55c5d82009e4 - test::test_main::hf2bc958b6a636d99
at src/libtest/lib.rs:120
34: 0x55c5d8201961 - test::test_main_static::h90326c29e77e672b
at src/libtest/lib.rs:139
35: 0x55c5d81de6b6 - test::main::h314b86d7399b1960
36: 0x55c5d81ddc00 - std::rt::lang_start::{{closure}}::hbaebdb534a364c8b
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/rt.rs:61
37: 0x55c5d8265fc3 - std::rt::lang_start_internal::{{closure}}::h040fbb64fb819fe8
at src/libstd/rt.rs:48
38: 0x55c5d8265fc3 - std::panicking::try::do_call::hfb8f62e9781fa17e
at src/libstd/panicking.rs:287
39: 0x55c5d826d1aa - __rust_maybe_catch_panic
at src/libpanic_unwind/lib.rs:79
40: 0x55c5d8266a7d - std::panicking::try::h2aa48346968c7185
at src/libstd/panicking.rs:265
41: 0x55c5d8266a7d - std::panic::catch_unwind::h8da95d989d5c8997
at src/libstd/panic.rs:396
42: 0x55c5d8266a7d - std::rt::lang_start_internal::h298ac725745c4eb6
at src/libstd/rt.rs:47
43: 0x55c5d81ddbd9 - std::rt::lang_start::hce40aa1f7d621390
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/rt.rs:61
44: 0x55c5d81de6ea - main
45: 0x7f5e423d1f2b - __libc_start_main
at ../csu/libc-start.c:308
46: 0x55c5d81dc18a - _start
47: 0x0 - <unknown>
---- roundtrip_lvl_9 stdout ----
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1165:5
stack backtrace:
0: 0x55c5d8263be4 - backtrace::backtrace::libunwind::trace::h93069dd45e563114
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
1: 0x55c5d8263be4 - backtrace::backtrace::trace_unsynchronized::ha81d63fb01c61f43
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
2: 0x55c5d8263be4 - std::sys_common::backtrace::_print_fmt::h5800670bd4856d97
at src/libstd/sys_common/backtrace.rs:77
3: 0x55c5d8263be4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8e952e7a864dce8d
at src/libstd/sys_common/backtrace.rs:61
4: 0x55c5d8286c5c - core::fmt::write::ha6cd837d898e4a61
at src/libcore/fmt/mod.rs:1030
5: 0x55c5d81e06d5 - std::io::Write::write_fmt::haa9265683e464bfb
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/io/mod.rs:1412
6: 0x55c5d825ea91 - std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt::he7c7a5301251df0f
at src/libstd/io/impls.rs:141
7: 0x55c5d8265e9e - std::sys_common::backtrace::_print::he810963362c39756
at src/libstd/sys_common/backtrace.rs:65
8: 0x55c5d8265e9e - std::sys_common::backtrace::print::hde04e6d280846a89
at src/libstd/sys_common/backtrace.rs:50
9: 0x55c5d8265e9e - std::panicking::default_hook::{{closure}}::h9d7792d5f2173a87
at src/libstd/panicking.rs:188
10: 0x55c5d8265b38 - std::panicking::default_hook::h9970cb21e5e5b2d4
at src/libstd/panicking.rs:202
11: 0x55c5d826659b - std::panicking::rust_panic_with_hook::hcbf43bd831dd8ef3
at src/libstd/panicking.rs:464
12: 0x55c5d826613e - std::panicking::continue_panic_fmt::h47b6e25372f76c95
at src/libstd/panicking.rs:373
13: 0x55c5d8266026 - rust_begin_unwind
at src/libstd/panicking.rs:302
14: 0x55c5d8283a7e - core::panicking::panic_fmt::h24ff570098569537
at src/libcore/panicking.rs:141
15: 0x55c5d8283b77 - core::result::unwrap_failed::hf7d72924a9a2d22c
at src/libcore/result.rs:1165
16: 0x55c5d81dd300 - core::result::Result<T,E>::unwrap::h0830fd1b36ee5052
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libcore/result.rs:933
17: 0x55c5d81ddcbc - test::get_test_file_data::h65896587825f1fc9
at tests/test.rs:11
18: 0x55c5d81dde04 - test::get_test_data::h65a752d85858b18c
at tests/test.rs:53
19: 0x55c5d81dde5c - test::roundtrip::h91d60028ff561479
at tests/test.rs:57
20: 0x55c5d81de67b - test::roundtrip_lvl_9::hbbb589037d403d50
at tests/test.rs:71
21: 0x55c5d81df09a - test::roundtrip_lvl_9::{{closure}}::hee71adf26a769dd9
at tests/test.rs:70
22: 0x55c5d81debde - core::ops::function::FnOnce::call_once::had14d0d8939888d8
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libcore/ops/function.rs:227
23: 0x55c5d81eb32f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h19a09fd199a0ff21
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/liballoc/boxed.rs:942
24: 0x55c5d826d1aa - __rust_maybe_catch_panic
at src/libpanic_unwind/lib.rs:79
25: 0x55c5d820690a - std::panicking::try::h1d1a1c5142fbfdd7
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/panicking.rs:265
26: 0x55c5d820690a - std::panic::catch_unwind::h200ca6210a82d128
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/panic.rs:396
27: 0x55c5d820690a - test::run_test_in_process::h766e5938de77f4da
at src/libtest/lib.rs:570
28: 0x55c5d820690a - test::run_test::run_test_inner::{{closure}}::h3d3ac74133b541d4
at src/libtest/lib.rs:473
29: 0x55c5d8205fe8 - test::run_test::run_test_inner::he77bc794d3ace3bd
at src/libtest/lib.rs:494
30: 0x55c5d8204569 - test::run_test::hba77973327a2fc7f
at src/libtest/lib.rs:530
31: 0x55c5d81fa954 - test::run_tests::h5a1270ddd5e0a473
at src/libtest/lib.rs:299
32: 0x55c5d81fa954 - test::console::run_tests_console::h29d755f301130a08
at src/libtest/console.rs:295
33: 0x55c5d82009e4 - test::test_main::hf2bc958b6a636d99
at src/libtest/lib.rs:120
34: 0x55c5d8201961 - test::test_main_static::h90326c29e77e672b
at src/libtest/lib.rs:139
35: 0x55c5d81de6b6 - test::main::h314b86d7399b1960
36: 0x55c5d81ddc00 - std::rt::lang_start::{{closure}}::hbaebdb534a364c8b
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/rt.rs:61
37: 0x55c5d8265fc3 - std::rt::lang_start_internal::{{closure}}::h040fbb64fb819fe8
at src/libstd/rt.rs:48
38: 0x55c5d8265fc3 - std::panicking::try::do_call::hfb8f62e9781fa17e
at src/libstd/panicking.rs:287
39: 0x55c5d826d1aa - __rust_maybe_catch_panic
at src/libpanic_unwind/lib.rs:79
40: 0x55c5d8266a7d - std::panicking::try::h2aa48346968c7185
at src/libstd/panicking.rs:265
41: 0x55c5d8266a7d - std::panic::catch_unwind::h8da95d989d5c8997
at src/libstd/panic.rs:396
42: 0x55c5d8266a7d - std::rt::lang_start_internal::h298ac725745c4eb6
at src/libstd/rt.rs:47
43: 0x55c5d81ddbd9 - std::rt::lang_start::hce40aa1f7d621390
at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/rt.rs:61
44: 0x55c5d81de6ea - main
45: 0x7f5e423d1f2b - __libc_start_main
at ../csu/libc-start.c:308
46: 0x55c5d81dc18a - _start
47: 0x0 - <unknown>
failures:
roundtrip_lvl_0
roundtrip_lvl_1
roundtrip_lvl_9
test result: FAILED. 4 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out
Right now in decompression we may use uninitialized memory without compiler-checked guarantees after with_init_state_only
, addressing arrays C-style. More rust-y way would be to encapsulate these unsafe operations in another library: smallvec. Implementation should be pretty straightforward.
In compression miniz_oxide always 0-initializes all buffers, so small tests are slow compared to miniz (which has valgrind warnings here). I am not sure how hard this part would be. Maybe we would be forced to split HashBuffers
's dict
buffer.
https://travis-ci.org/Frommi/miniz_oxide/builds/267227423
It seems that everything builds properly, all the folders contain right files, but
../bin/miniz_tester -v a linux-4.8.11
doesn't find the files:
pwd
output:
/home/travis/build/Frommi/miniz_oxide/test_scratch
so we are in the right folder.
ls
output:
linux-4.8.11 linux-4.8.11.tar.xz
ls linux-4.8.11
output:
arch crypto include kernel net security
block Documentation init lib README sound
certs drivers ipc MAINTAINERS REPORTING-BUGS tools
COPYING firmware Kbuild Makefile samples usr
CREDITS fs Kconfig mm scripts virt
so linux is downloaded and extracted properly.
ls ../bin
output:
example1 example2 example3 example4 example5 example6 miniz_tester
so miniz_tester
is compiled and then it is executed ../bin/miniz_tester -v a linux-4.8.11
, but can't detect files:
Total files processed: 0
In miniz deflation usually consists of one tdefl_init
and multiple tdefl_compress
. Both calls carry pointers/references to data which is alive for different time, but in C is stored in one struct.
So I divided the struct in two: CallbackOxide
for one tdefl_compress
call, responsible for input buffers and output buffers and the rest in CompressorOxide
.
But CallbackOut
is a enum: it can be a function or a buffer and function is supposed to live as long as CompressorOxide
and is given at tdefl_init
(or not). So, I need to extract this Option
of callback function and place it into CallbackOut
which is unwieldy.
I mostly preserved all functions, but divided huge "global-struct" tdefl_compressor
from C into sub-structs by meaning, so functions now carry a bundle of these structs. It is better, but now I must carry a list.
There is a mess of error codes, especially in high-level functions: like so.
write in Cursor takes a slice and so sometimes it produces something like this:
while rle.rle_repeat_count != 0 {
rle.rle_repeat_count -= 1;
packed_code_sizes.write(&[rle.prev_code_size][..])?;
}
There is an option to make a vec of size rle.rle_repeat_count
, but it will be on heap and probably slow.
Hottest function seems to be this one.
This would help those who scrutinize changes before upgrading their dependencies.
No need to backfill the changelog with the past history: a changelog that starts now on is still way better than nothing.
I can add a changelog template if you like.
Would it be okay with some slight differences in output from the compressor compared to miniz?
E.g I think it would be interesting to attempt implementing this matching algorithm as it looks like it could give significant performance improvements at higher compression levels.
There are also some minor compression level improvements that we could make, miniz doesn't do matches with the first byte, and does a very simple block type selection that doesn't always produce great results on very short input. (As opposed to zlib.)
Should we keep the compressor/decompressor structs exactly the same, or is it enough that inflate/deflateinit etc behave the same way?
I also propose we separate the C api part(the extern c exports) as a separate wrapper crate to avoid possible symbol conflicts when using it as a rust library and don't have to create the c static lib when we don't need it.
When trying to decompress the following data using this Rust code:
use miniz_oxide::inflate::decompress_to_vec;
fn main() {
let data = [31, 139, 8, 0, 0, 0, 0, 0, 0, 3, 149, 147, 61, 75, 195, 80, 20, 134, 79, 62, 20, 84, 170, 1, 17, 28, 68, 28, 132, 110, 183, 247, 38, 185, 249, 154, 58, 10, 130, 116, 116, 170, 54, 109, 18, 11, 173, 169, 109, 90, 112, 210, 209, 81, 112, 112, 22, 127, 131, 56, 232, 232, 224, 143, 112, 112, 238, 159, 168, 55, 31, 234, 77, 91, 82, 26, 184, 57, 36, 79, 222, 115, 222, 251, 38, 9, 170, 27, 218, 209, 62, 128, 113, 76, 234, 181, 83, 82, 55, 31, 175, 223, 101, 0, 216, 249, 120, 217, 219, 102, 181, 244, 244, 186, 203, 10, 8, 108, 109, 18, 221, 70, 132, 234, 136, 152, 20, 81, 13, 222, 132, 148, 43, 11, 184, 144, 241, 178, 138, 49, 113, 176, 171, 90, 142, 175, 106, 45, 199, 79, 46, 217, 145, 63, 53, 126, 117, 241, 92, 49, 215, 215, 48, 17, 197, 185, 185, 179, 156, 252, 113, 49, 227, 91, 60, 39, 148, 240, 190, 196, 127, 95, 134, 217, 116, 176, 238, 89, 177, 47, 181, 200, 151, 180, 156, 206, 229, 247, 35, 229, 252, 176, 156, 8, 198, 252, 126, 138, 184, 144, 241, 57, 57, 106, 139, 114, 148, 167, 115, 178, 73, 46, 199, 34, 46, 100, 124, 206, 126, 245, 162, 185, 98, 166, 227, 242, 55, 16, 81, 49, 159, 227, 18, 125, 93, 222, 207, 202, 76, 14, 126, 172, 163, 69, 126, 148, 76, 87, 178, 9, 139, 213, 66, 148, 185, 177, 48, 0, 159, 211, 52, 167, 118, 198, 27, 189, 145, 134, 6, 145, 215, 65, 205, 176, 11, 240, 201, 158, 171, 150, 36, 104, 177, 90, 211, 37, 184, 99, 63, 11, 30, 2, 124, 63, 200, 73, 253, 98, 141, 206, 199, 233, 119, 18, 63, 11, 207, 34, 76, 38, 147, 155, 120, 193, 248, 48, 185, 23, 207, 186, 117, 214, 146, 26, 247, 57, 56, 1, 184, 63, 19, 18, 189, 82, 102, 51, 47, 162, 168, 55, 112, 42, 149, 8, 117, 189, 10, 123, 247, 65, 219, 95, 247, 195, 97, 127, 112, 53, 108, 244, 61, 144, 221, 246, 101, 192, 116, 171, 65, 24, 6, 29, 47, 13, 83, 73, 203, 15, 58, 186, 13, 141, 216, 3, 0, 0];
assert!(decompress_to_vec(&data).is_ok());
}
I'm hitting the following error:
Err(TINFLStatus::Failed)
Python's gzip
can decompress it just fine, though:
import gzip
data = [31, 139, 8, 0, 0, 0, 0, 0, 0, 3, 149, 147, 61, 75, 195, 80, 20, 134, 79, 62, 20, 84, 170, 1, 17, 28, 68, 28, 132, 110, 183, 247, 38, 185, 249, 154, 58, 10, 130, 116, 116, 170, 54, 109, 18, 11, 173, 169, 109, 90, 112, 210, 209, 81, 112, 112, 22, 127, 131, 56, 232, 232, 224, 143, 112, 112, 238, 159, 168, 55, 31, 234, 77, 91, 82, 26, 184, 57, 36, 79, 222, 115, 222, 251, 38, 9, 170, 27, 218, 209, 62, 128, 113, 76, 234, 181, 83, 82, 55, 31, 175, 223, 101, 0, 216, 249, 120, 217, 219, 102, 181, 244, 244, 186, 203, 10, 8, 108, 109, 18, 221, 70, 132, 234, 136, 152, 20, 81, 13, 222, 132, 148, 43, 11, 184, 144, 241, 178, 138, 49, 113, 176, 171, 90, 142, 175, 106, 45, 199, 79, 46, 217, 145, 63, 53, 126, 117, 241, 92, 49, 215, 215, 48, 17, 197, 185, 185, 179, 156, 252, 113, 49, 227, 91, 60, 39, 148, 240, 190, 196, 127, 95, 134, 217, 116, 176, 238, 89, 177, 47, 181, 200, 151, 180, 156, 206, 229, 247, 35, 229, 252, 176, 156, 8, 198, 252, 126, 138, 184, 144, 241, 57, 57, 106, 139, 114, 148, 167, 115, 178, 73, 46, 199, 34, 46, 100, 124, 206, 126, 245, 162, 185, 98, 166, 227, 242, 55, 16, 81, 49, 159, 227, 18, 125, 93, 222, 207, 202, 76, 14, 126, 172, 163, 69, 126, 148, 76, 87, 178, 9, 139, 213, 66, 148, 185, 177, 48, 0, 159, 211, 52, 167, 118, 198, 27, 189, 145, 134, 6, 145, 215, 65, 205, 176, 11, 240, 201, 158, 171, 150, 36, 104, 177, 90, 211, 37, 184, 99, 63, 11, 30, 2, 124, 63, 200, 73, 253, 98, 141, 206, 199, 233, 119, 18, 63, 11, 207, 34, 76, 38, 147, 155, 120, 193, 248, 48, 185, 23, 207, 186, 117, 214, 146, 26, 247, 57, 56, 1, 184, 63, 19, 18, 189, 82, 102, 51, 47, 162, 168, 55, 112, 42, 149, 8, 117, 189, 10, 123, 247, 65, 219, 95, 247, 195, 97, 127, 112, 53, 108, 244, 61, 144, 221, 246, 101, 192, 116, 171, 65, 24, 6, 29, 47, 13, 83, 73, 203, 15, 58, 186, 13, 141, 216, 3, 0, 0]
assert gzip.decompress(bytes(data)) is not None
I presume this is the same as sile/libflate#45, which I'm linking because it provides a clearer error message which may be helpful:
btype 0x11 of DEFLATE is reserved(error) value
I'm not sure if this bug should be filled here, but I guess it can't hurt. Is there any way to make this library decompress the data successfully, or a way to work around it?
I have also tested flate2
as mentioned in the README, and that's able to decompress the data too.
Hi, I have wrapped the streaming (ring-buffer using) API of miniz_oxide behind a higher level API here: https://github.com/golddranks/stream_zipper/blob/master/src/deflate.rs
Is there any interest in supporting such a higher level API within miniz_oxide itself?
I've experimented with a state machine-like API:
pub enum State<'a> {
HasMoreOutput(ChunkIter<'a>),
NeedsMoreInput(InputSink),
Stop(FinishedStream<'a>),
}
The idea is that every function that operates on the state takes self as value, and as such, parsing the stream can proceed as a state machine alternating between states "HasMoreOutput" and "NeedsMoreInput". The states own the internal state, including the ring buffer, which makes this a higher level API as the one provided by miniz_oxide at the moment.
Here's a roughly how it would be used:
let mut state = start_deflate_stream();
for chunk in pure_deflate_stream.chunks(500) {
if let State::NeedsMoreInput(sink) = state {
state = sink.feed(chunk)?;
};
while let State::HasMoreOutput(out) = state {
println!("Got {:?}", out.get());
state = out.next()?;
}
}
Another experiment I did was to have just InputSink
, and then passing a callback that gets the uncompressed output. This is a kind of an internal iterator. Again, here's an example
let mut state = start_deflate_stream();
for chunk in pure_deflate_stream.chunks(50) {
if let State::NeedsMoreInput(sink) = state {
state = sink.inner_iter(chunk, |out| {
println!("Got {:?}", out);
})?;
};
}
What do you think about introducing such a high level API?
Continuing from #29: inflate::decompress_to_vec()
has its own fuzzing target and is covered pretty well. There should also be a fuzzing target for the streaming API.
When decoding lightly compressed data the majority of the time is spent in adler32::RollingAdler32::update_buffer
, so even small speedups in it will translate to measurable speeupds in the entire decoding.
Profiling results from miniz-oxide
branch of png
crate to back this up: https://perfht.ml/2V3ml3q
On Fedora, we found that 0.2.4 broke for i686 and armv7hl, such as this build.
error[E0308]: mismatched types
--> src/lib.rs:252:79
|
252 | as_c_return_code(mz_compress2_oxide(&mut stream_oxide, level, dest_len))
| ^^^^^^^^ expected `u64`, found `u32`
|
= note: expected mutable reference `&mut u64`
found mutable reference `&mut u32`
error[E0308]: mismatched types
--> src/lib.rs:291:74
|
291 | as_c_return_code(mz_uncompress2_oxide(&mut stream_oxide, dest_len))
| ^^^^^^^^ expected `u64`, found `u32`
|
= note: expected mutable reference `&mut u64`
found mutable reference `&mut u32`
error[E0308]: mismatched types
--> src/c_export.rs:150:23
|
150 | total_in: self.total_in,
| ^^^^^^^^^^^^^ expected `u32`, found `u64`
error[E0308]: mismatched types
--> src/c_export.rs:160:24
|
160 | total_out: self.total_out,
| ^^^^^^^^^^^^^^ expected `u32`, found `u64`
error[E0308]: mismatched types
--> src/c_export.rs:211:23
|
211 | total_in: stream.total_in,
| ^^^^^^^^^^^^^^^ expected `u64`, found `u32`
error[E0308]: mismatched types
--> src/c_export.rs:213:24
|
213 | total_out: stream.total_out,
| ^^^^^^^^^^^^^^^^ expected `u64`, found `u32`
error: aborting due to 6 previous errors
For more information about this error, try `rustc --explain E0308`.
error: could not compile `miniz_oxide_c_api`.
I bisected the regression to commit 918aa73. For the mz_uncompress2_oxide
errors, that parameter changed from &mut c_ulong
to &mut u64
. StreamOxide
's total_in
and total_out
also changed from c_ulong
to u64
. Those are the same type on LP64 targets (most 64-bit Unix), but not on 32-bit targets, nor 64-bit Windows (LLP64).
See also #54.
There are still a number of uses of unsafe in the encoder code, more specifically in the match finding code to avoid bounds checks inside tight loops. While we've looked over them before, it would be a good idea to check them again and try to get rid of as many as we can if they have no or negligible effect on performance. In many of the unsafe calls the compiler may already be able to optimize them away anyhow.
The only unsafe code in the decoder are simple bytes to integer conversions, which should be sound.
Once 1.34 is reliably available in distros, we can switch to using the built-in functions for converting from bytes to integers.
miniz_oxide::inflate::core::decompress
will panic if a output buffer of size 0 is passed in. The panic is due to an overflow in core.rs, on line 1020: out_cur.get_ref().len() - 1
.
It says that "This function shouldn't panic pending any bugs." in the documentation so I presume this is a bug.
I was looking into a build failure on CI and noticed that a build command is panicking when trying to extract a zip archive. It seems to be failing at this assertion.
Looking at the diff that introduced this change, I found the following comment:
2a5703a#diff-1b531c5e405e0bc95890d52cf2f2f5b5R44
zip is broken right now due to struct difference between c and rust version
That makes it sound like ZIP is not supported in the latest release. Is this correct? In a non-breaking release this is likely to break a lot of things downstream, which will only be noticed at runtime.
Downgrading to 0.3.3 fixes this problem.
Just wondering if you'd be interested in adding the Zlib license as a license option for this crate.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.