mirage / decompress Goto Github PK
View Code? Open in Web Editor NEWPure OCaml implementation of Zlib.
License: MIT License
Pure OCaml implementation of Zlib.
License: MIT License
Like it is possible with "Zlib_deflate.default", like Python zlib allows it: https://docs.python.org/3/library/zlib.html
It is gaining popularity and it seems that OCaml bindings are abandoned: https://github.com/ygrek/ocaml-zstd
See format description https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md
And decoding testing tool https://github.com/facebook/zstd/tree/v1.3.4/tests#decodecorpus---tool-to-generate-zstandard-frames-for-decoder-testing
It's only about the header, so (I think), it's easy to implement this (@juanchanco did already this feature but in the old version of Decompress).
This issue is not for the next version of Decompress (1.0).
CC: @yomimono, @Engil, @verbosemode
Any plans for this? I just updated the inflator.ml to mirror Canopy's to make this work for the Syslog project: verbosemode/syslogd-mirage#9
Wondering if there is a better way we can do this other than copying files?
The test codes have a different objective (and I already had a look at them).
Currently, deflate
algorithm use an Oakasaki's queue and it could be more interesting to find a data-structure which concatenation is fast with same semantic of Queue.
Then, in the blocklz
algorithm, we use the stdlib's queue which is a double linked-list. From some benchmark in some other works (encore
and faraday
), the stdlib's is not the best choice in some contexts. So we need to figure about that (context & benchmark).
This issue is a starter to optimize (again) decompress. However, we already found the bottleneck - the adler-32 checksum.
It would be really nice if you could add a Dockerfile
and automated build on Docker Hub so I could easily try this out without having to build it.
@avsm has a nice set of Ocaml base containers which make it easier.
With the following git pack:
http://tumbolandia.net/pack.dump
I get Exception: Decompress_inflate.Make(X).Invalid_crc.
, even though this seems to be perfectly fine for Zlib.
I used @yomimono's interface of Inflator available here: https://github.com/Engil/Canopy/blob/master/inflator.ml, and down here you can find a session of utop that reproduces the error.
───────────────────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────
│ Welcome to utop version 1.18.2 (using OCaml version 4.02.3)! │
└──────────────────────────────────────────────────────────────┘
Type #utop_help for help about using utop.
─( 19:18:02 )─< command 0 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # #require "irmin";;
─( 19:18:02 )─< command 1 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # #require "decompress";;
─( 19:18:11 )─< command 2 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # #require "git.unix";;
─( 19:18:17 )─< command 3 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # let load_file f =
let ic = open_in f in
let n = in_channel_length ic in
let s = Bytes.create n in
really_input ic s 0 n;
close_in ic;
(s);;
val load_file : bytes -> bytes = <fun>
─( 19:18:22 )─< command 4 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # let m =
let b = load_file "/tmp/pack.dump" in
Mstruct.of_string b;;
val m : Mstruct.t = <abstr>
─( 19:18:30 )─< command 5 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # #mod_use "Canopy/inflator.ml";;
module Inflator :
sig
module IDBytes :
sig
external length : bytes -> int = "%string_length"
external get : bytes -> int -> char = "%string_safe_get"
external set : bytes -> int -> char -> unit = "%string_safe_set"
external create : int -> bytes = "caml_create_string"
val make : int -> char -> bytes
val init : int -> (int -> char) -> bytes
val empty : bytes
val copy : bytes -> bytes
val of_string : bytes -> bytes
val to_string : bytes -> bytes
val sub : bytes -> int -> int -> bytes
val sub_string : bytes -> int -> int -> bytes
val extend : bytes -> int -> int -> bytes
val fill : bytes -> int -> int -> char -> unit
val blit : bytes -> int -> bytes -> int -> int -> unit
val blit_string : bytes -> int -> bytes -> int -> int -> unit
val cat : bytes -> bytes -> bytes
val iter : (char -> unit) -> bytes -> unit
val iteri : (int -> char -> unit) -> bytes -> unit
val map : (char -> char) -> bytes -> bytes
val mapi : (int -> char -> char) -> bytes -> bytes
val trim : bytes -> bytes
val escaped : bytes -> bytes
val index : bytes -> char -> int
val rindex : bytes -> char -> int
val index_from : bytes -> int -> char -> int
val rindex_from : bytes -> int -> char -> int
val contains : bytes -> char -> bool
val contains_from : bytes -> int -> char -> bool
val rcontains_from : bytes -> int -> char -> bool
val uppercase : bytes -> bytes
val lowercase : bytes -> bytes
val capitalize : bytes -> bytes
val uncapitalize : bytes -> bytes
type t = bytes
val compare : t -> t -> int
val unsafe_to_string : t -> t
val unsafe_of_string : t -> t
external unsafe_get : t -> int -> char = "%string_unsafe_get"
external unsafe_set : t -> int -> char -> unit = "%string_unsafe_set"
external unsafe_blit : t -> int -> t -> int -> int -> unit = "caml_blit_string" "noalloc"
external unsafe_fill : t -> int -> int -> char -> unit = "caml_fill_string" "noalloc"
val concat : t -> t list -> t
val of_bytes : 'a -> 'a
val to_bytes : 'a -> 'a
end
module XInflator :
sig
type t = Decompress_inflate.Make(IDBytes).t
type src = [ `Channel of in_channel | `Manual of int -> bytes | `String of int * bytes ]
type dst = bytes
val make : [< src ] -> dst -> t
val eval : t -> [ `Error | `Flush of int | `Ok of int ]
val contents : t -> int
val flush : t -> unit
end
module XDeflator :
sig
type t = Decompress_deflate.Make(IDBytes).t
type src = [ `Channel of in_channel | `Manual of unit -> bytes * bool | `String of int * bytes ]
type dst = bytes
val make : ?window_bits:int -> [< src ] -> dst -> t
val eval : t -> [ `Error | `Flush | `Ok ]
val contents : t -> int
val flush : t -> unit
end
val inflate : ?output_size:int -> Mstruct.t -> Mstruct.t option
val deflate : ?level:'a -> Cstruct.t -> Cstruct.t
end
─( 19:18:33 )─< command 6 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # Inflator.inflate (Mstruct.clone m);;
output_size: 0 wrote 416217 of 416217
Exception: Decompress_inflate.Make(X).Invalid_crc.
─( 19:18:46 )─< command 7 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # Git_unix.Zlib.inflate (Mstruct.clone m);;
- : Mstruct.t option = Some <abstr>```
Array1.map_file
was removed in 4.08.0
See ocaml/opam-repository@0230f2f
I have this code
(* Data is a Bigarray.Array1 of chars *)
(* FIXME: For some reason it eats memory like crazy on one sample *)
let unzlib_raw ba =
let open Decompress in
Printf.printf "ZLib: Before unpack\n";
let data = Util.ba_to_bytes ba in
let datalen = Bytes.length data in
Printf.printf "ZLib: datalen = 0x%x\n" datalen;
Out_channel.flush Out_channel.stdout;
let inputbuf = Bytes.create 0xFFFF in
let outputbuf = Bytes.create 0xFFFF in
let window = Window.create ~proof:B.proof_bytes in
let pos = ref 0 in
let res = Buffer.create datalen in
Zlib_inflate.bytes inputbuf outputbuf
(fun inputbuf ->
let n = min 0xFFFF (datalen - !pos) in
(* TODO: More efficient algorithm without copy *)
Bytes.blit data !pos inputbuf 0 n;
pos := !pos + n;
n)
(fun outputbuf len ->
Printf.printf "ZLib: outfun len = 0x%x\n" len;
Out_channel.flush Out_channel.stdout;
Buffer.add_subbytes res outputbuf 0 len;
0xFFFF)
(Zlib_inflate.default window)
|> function
| Ok _ ->
Printf.printf "ZLib: Unpack successful\n";
Some (Buffer.contents res)
| Error e ->
Printf.printf "ZLib: Unpack failure\n";
None
And I have a chunk of the data (it might be not a valid Zlib data, but it should still return a failure then, instead of perpetuating and eating all of memory until it gets OOM-killed. Attaching a ZIP archive of it (note it is not the ZIP causing problems, it is faulty_data.bin
file inside)
faulty_data.zip
Since it seems that ocaml-lzma is dead https://forge.ocamlcore.org/projects/ocaml-lzma/ and there are no libraries that provide a way to do that, it make sense to support it here too.
Gzip support seems to have gone missing in the 1.0 release.
I was unable to build the benchmark in the way mentioned in the Benchmark section of the readme.
I was getting the following error when I tried to run it
$ dune build bench/output.csv
Error: Don't know how to build bench/output.csv
My current opam switch is 4.12.1
and I do have dune
installed locally
Any help on this would be great! 😃
In some context such as Git, we are able to store a full zlib
contents into one and unique buffer - when we manipulate small objects. Even if the stream API is interesting and useful for unknowable contents, we should have the opportunity to inflate a buffer and produce the output or fail such as:
val inflate : src:bigstring -> dst:bigstring -> (int * int, [> error ]) result
(** [inflate ~src ~dst] tries to inflate [src] and store the output into [dst]. It returns
byte(s) consumed into [src] and byte(s) produced into [dst]. *)
Such implementation should be outside the current de.ml
implementation and should just care about: we can inflate src
or we will fail. This is what libdeflate
does and, from this perspective, the constraint to save the world/state for a full-stream API disappears and we should be able to provide a fast inflation algorithm.
Mechanically, and due to the fact that ocaml-git
/Git has such heuristic - it tries to inflate in one go -, I think we will improve performance a lot where most of Git objects are small enough to fit under a IO_BUFFER_SIZE
buffer.
When I try to serialize the PACK file for ocaml-git
, I find a deep bug in the Inflate
compute.
The file: https://paste.isomorphis.me/Aka
The Deflate
computes without error the file (and zlib
can inflate the result withtout error). So the problem is about the Inflate
in decompress
. It's too late in Phnom Penh, I will check what is going on tomorrow.
For reviewer, de.ml
is a complex code base, I should explain some parts of it.
After updating decompress project using decompress library fails to build with:
File "_none_", line 1:
Error: No implementations provided for the following modules:
Checkseum referenced from /home/akochkov/.opam/default/lib/decompress/impl/decompress_impl.cmxa(Decompress_impl__Decompress_safe),
/home/akochkov/.opam/default/lib/decompress/impl/decompress_impl.cmxa(Decompress_impl)
~proof
should be replaced by ~witness
in the following comment, right?
Line 186 in 3778626
I had some trouble using this library since the higher-level interface is undocumented and still too-low-level for what I'm trying to do (decompress some data that I have in-memory).
I eventually came up with:
let inflate_string ?(buffer_size_bytes = 100000) data =
let input = Bigstring.of_string data
and buffer = Bigstring.create buffer_size_bytes
and output = Queue.create () in
let refill s = Bigstring.length s
and flush s n = Queue.enqueue output (Bigstring.sub s ~pos:0 ~len:n) in
Gz.Higher.uncompress ~refill ~flush input buffer
|> Result.map ~f:(fun _ ->
Queue.to_list output |> Bigstring.concat |> Bigstring.to_string)
Would it make sense to include something like this in the code, or at least give an example in the documentation?
Attaching is an archive of the input data buffer, that might be not a valid zlib data, but raise an exception.
Uncaught exception:
(Invalid_argument "index out of bounds")
Raised by primitive operation at file "lib/de.ml", line 1010, characters 20-38
Called from file "lib/de.ml", line 1319, characters 27-37
Called from file "lib/gz.ml", line 450, characters 12-31
Called from file "lib/gz.ml" (inlined), line 553, characters 17-22
Brotli is supported by most browsers as a content-encoding that compressed better than gzip and deflate. See:
The current implementation of the GZIP
interface exposes a compress
function. However, the encoder does not handle any compression level (see Gz.Def.encoder
interface documentation).
Is it planned to fully implement compression features through this library?
Outside the scope, decompress
does not compile because we catch a warning
as an error about deprecated re
functions when we test it.
This isn't a blocker for me, just thought you might want to know!
$ opam install git
The following actions will be performed:
∗ install decompress 0.7 [required by git]
∗ install git 1.11.5
===== ∗ 2 =====
Do you want to continue? [Y/n] y
<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><> 🐫
[decompress.0.7] found in cache
[git.1.11.5] found in cache
<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><> 🐫
[ERROR] The compilation of decompress failed at "/Users/ec/.opam/opam-init/hooks/opam-bin-cache.sh wrap
0c7e7b06c487aeb762d2892b750061fb27621af6a0b292f1c9c4f901cf91b582 ocaml pkg/pkg.ml build --pinned false --tests
false --with-cmdliner false".
#=== ERROR while compiling decompress.0.7 =====================================#
# context 2.0.0 | macos/x86_64 | ocaml-variants.4.07.0+flambda | https://opam.ocaml.org/#6d9b8fd1
# path ~/Sync/Code/gitlabel-parser/_opam/.opam-switch/build/decompress.0.7
# command ~/.opam/opam-init/hooks/opam-bin-cache.sh wrap 0c7e7b06c487aeb762d2892b750061fb27621af6a0b292f1c9c4f901cf91b582 ocaml pkg/pkg.ml build --pinned false --tests false --with-cmdliner false
# exit-code 1
# env-file ~/.opam/log/decompress-69483-e7966f.env
# output-file ~/.opam/log/decompress-69483-e7966f.out
### output ###
# [...]
# ocamlfind ocamlopt unix.cmxa -I /Users/ec/.opam/default/lib/ocamlbuild /Users/ec/.opam/default/lib/ocamlbuild/ocamlbuildlib.cmxa -linkpkg myocamlbuild.ml /Users/ec/.opam/default/lib/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
# + ocamlfind ocamlopt unix.cmxa -I /Users/ec/.opam/default/lib/ocamlbuild /Users/ec/.opam/default/lib/ocamlbuild/ocamlbuildlib.cmxa -linkpkg myocamlbuild.ml /Users/ec/.opam/default/lib/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
# File "myocamlbuild.ml", line 1:
# Error: Files /Users/ec/.opam/default/lib/ocamlbuild/ocamlbuildlib.cmxa
# and /Users/ec/Sync/Code/gitlabel-parser/_opam/lib/ocaml/stdlib.cmxa
# make inconsistent assumptions over implementation Stdlib__sys
# Command exited with code 2.
# pkg.ml: [ERROR] cmd ['ocamlbuild' '-use-ocamlfind' '-classic-display' '-j' '4' '-tag' 'debug'
# '-build-dir' '_build' 'opam' 'pkg/META' 'CHANGES.md' 'LICENSE.md'
# 'README.md' 'pkg/META' 'README.md' 'CHANGES.md' 'lib/decompress.a'
# 'lib/decompress.cmxs' 'lib/decompress.cmxa' 'lib/decompress.cma'
# 'lib/decompress.cmx' 'lib/decompress.cmi' 'lib/decompress.mli']: exited with 10
When I build the tests with NixOS, the following happens. Reproducible with 4.10, 4.11 and 4.12.
File "bindings/stubs/dune", line 15, characters 22-26:
15 | (forbidden_libraries unix)
^^^^
Error: Library "unix" was pulled in.
-> required by library "bigarray" in
/nix/store/56bmf5x8l90znfdvigxbj92iwzdyrva9-ocaml-4.11.2/lib/ocaml
-> required by library "ctypes" in
/nix/store/l91x7cq89449y7xcj1sq8645nkkmh6vh-ocaml4.11.2-ctypes-0.17.1/lib/ocaml/4.11.2/site-lib/ctypes
-> required by library "ctypes.stubs" in
/nix/store/l91x7cq89449y7xcj1sq8645nkkmh6vh-ocaml4.11.2-ctypes-0.17.1/lib/ocaml/4.11.2/site-lib/ctypes
-> required by library "gen_decompress_bindings" in
_build/default/bindings/stubs
test alias test/runtest
Patching this line out makes the tests pass without a hitch. Is this safe to do?
We can implement (as zlib) a flush method (sync, partial or full) in Decompress. We need more test about that.
A simple optimization should be to use get_int16_be
(le
?) here:
Lines 999 to 1004 in 366ff8d
The loop already ensures to have, at least 2 bytes available - so we can use an unsafe
operation. However, we must prove that it's a real optimization - so we can use dinosaure/lzbench.
BGZF is a GZIP-based block compression format widely used in bioinformatics. BGZF uses extra fields as specified in RFC 1952 and is designed to be compatible with GZIP decompressors.
The Gz.Inf
module decompressor fails to read data from a BGZF file. The MWE with its input file:
$ decompress --format gzip mwe.txt.gz
decompress: Unexpected end of input.
I wonder why the current GZIP decompressor implementation fails to read BGZF and whether it can be easily fixed.
The input file mwe.txt.gz was created in the following way using the bgzip tool:
$ seq 1 10 > mwe.txt
$ bgzip mwe.txt
I have a hit a buffer overflow in De.Inf.Ns.inflate
. This has ultimately come from a call through ocaml-git decompressing a git pack, so it's not immediately clear to me what the input is at this point. I can capture it if you need it though.
The code in question is after line 1600 in lib/de.ml:
let inflate lit dist d =
try
let rec inflate_loop () =
__fill_bits d lit.Lookup.l
; let value =
lit.Lookup.t.(d.hold land lit.Lookup.m) land Lookup.mask in
let len = lit.Lookup.t.(d.hold land lit.Lookup.m) lsr 15 in
d.hold <- d.hold lsr len
; d.bits <- d.bits - len
; if value < 256 then (
unsafe_set_uint8 d.o d.o_pos value
; d.o_pos <- d.o_pos + 1
; inflate_loop ())
else if value == 256 then raise_notrace End
else
let l = value - 257 in
The failure is an Invalid_argument from that unsafe_set_uint8
. d.o has length 64k, and d.o_pos is 64k. My tracing shows that we've been around this loop filling up the buffer, and obviously this is the point that we've fallen off the end.
I don't have any clue what this code is doing, so I'm at a loss and would appreciate some assistance. I can see though that d.o_pos
is being incremented around this loop, and there's no bounds checking that I can see, so I hope someone more knowledgeable than me could understand what's going on here.
Tagging @dinosaure in light of #100 :-)
Still uses old functions and module names, some are already incompatible.
zlib
wraps only the RFC1951 standard - so we already implemented it. It could be interesting to export only the RFC1951 decoder/encoder (in a smart way ...) and avoid the zlib
header and the Adler-32 checksum at the end.
I am trying to build decompress.1.1.0 using dune.2.5.1 for Multicore OCaml 4.10.0. It builds and installs fine, but, when I try to use it using:
(libraries checkseum.ocaml decompress)
I get the following error:
File "benchmarks/decompress/dune", line 4, characters 29-39:
4 | (libraries checkseum.ocaml decompress))
^^^^^^^^^^
Error: Library "decompress" not found.
Source: https://github.com/ocaml-bench/sandmark/blob/master/benchmarks/decompress/dune
The above works fine with decompress.0.8.1 with dune-2.5.1 and Multicore OCaml 4.10.0 though! As mentioned in the README for issue #47, I then tried the following syntax:
(libraries (checkseum.ocaml decompress)))
But, I get the following error:
File "benchmarks/decompress/dune", line 4, characters 13-41:
4 | (libraries (checkseum.ocaml decompress)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Atom or quoted string expected
How can this be resolved?
I am not sure if this checkseum build problem. A few days I built decompress just fine, but today something got broken.
The system is Fedora 33 x86_64
[i] ℤ gcc --version
gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1)
[i] ℤ ld --version
GNU ld version 2.35-18.fc33
[i] ℤ ocamlc --version
4.11.1
[i] ℤ opam --version
2.0.6
Might be useful to also trigger then the checkseum
rebuild somehow if/when decompress
is updated.
[i] ℤ opam upgrade 11:51:41
The following actions will be performed:
↻ recompile decompress 1.4.0 [upstream changes]
Do you want to continue? [Y/n] Y
<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[decompress.1.4.0] found in cache
<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[ERROR] The compilation of decompress failed at "/home/akochkov/.opam/opam-init/hooks/sandbox.sh build dune build -p decompress -j 7".
#=== ERROR while compiling decompress.1.4.0 ===================================#
# context 2.0.6 | linux/x86_64 | ocaml-system.4.11.1 | https://opam.ocaml.org#3b10ec4f
# path ~/.opam/default/.opam-switch/build/decompress.1.4.0
# command ~/.opam/opam-init/hooks/sandbox.sh build dune build -p decompress -j 7
# exit-code 1
# env-file ~/.opam/log/decompress-960269-e7966f.env
# output-file ~/.opam/log/decompress-960269-e7966f.out
### output ###
# ocamlopt bin/pipe.exe (exit 2)
# (cd _build/default && /usr/bin/ocamlopt.opt -w -40 -g -o bin/pipe.exe /home/akochkov/.opam/default/lib/bigarray-compat/bigarray_compat.cmxa /home/akochkov/.opam/default/lib/optint/optint.cmxa /home/akochkov/.opam/default/lib/checkseum/c/checkseum_c.cmxa -I /home/akochkov/.opam/default/lib/checkseum/c /usr/lib64/ocaml/unix.cmxa -I /usr/lib64/ocaml lib/de.cmxa lib/zl.cmxa lib/gz.cmxa /home/akoc[...]
# lto1: fatal error: bytecode stream in file ‘/home/akochkov/.opam/default/lib/checkseum/c/libcheckseum_c_stubs.a’ generated with LTO version 9.2 instead of the expected 9.4
# compilation terminated.
# lto-wrapper: fatal error: gcc returned 1 exit status
# compilation terminated.
# /usr/bin/ld: error: lto-wrapper failed
# collect2: error: ld returned 1 exit status
# File "caml_startup", line 1:
# Error: Error during linking (exit code 1)
<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build decompress 1.4.0
└─
╶─ No changes have been performed
~/data/ocaml took 6s
Can I use that library now or should I wait a little bit longer? :-)
Like xmlm to use Decompress in C++ benchmark.
Just so we know we agree with foreign implementations :)
Here are some:
I wasn't able to find a set of official test vectors, but perhaps someone with stronger google-fu than me can be of assistance.
With alcotest
. I love colors in my terminal.
Now that we have a workaround for the "linking" hack: https://github.com/diml/variants-workaround it would be great to switch decompress to dune!
Formats like GZip do not define the end marker of the stream, and doing actual decompression for just finding out the size is overkill. Is it possible to do a "lighter" version of the walking algorithm to find the end of compressed data? And export this as an API.
I have a dream, a dream with Decompress was 10 times faster than the implementation of C. Maybe it's not just a dream.
Hello, I'm using decompress to perform Gz
streaming compression of some ascii log. When killing my soft, I've a cleanup function that tries to gracefully empty up the internal buffers of decompress
. Note that the gzip file produced was rather large at this stage (roughly 650 megabytes).
I noticed the following exception was raised:
Invalid_argument
"Optint.to_int32: 13903751204 can not fit into a 32 bits integer")
and indeed my gzip file is truncated at the end (checked using gzip -t
), meaning that there was an error during this flushing down. Here's my flushing function (mostly inspired by the code for the pipe utility as you might notice):
let rec flush_encoder oc encoder =
let open Gz in
match Def.encode encoder with
| `Await encoder ->
(* A length of 0 signals end of input (I think). *)
flush_encoder oc (Def.src encoder i 0 0)
| `Flush encoder ->
let len = io_buffer_size - Def.dst_rem encoder in
bigstring_output oc o 0 len ;
flush_encoder oc (Def.dst encoder o 0 io_buffer_size)
| `End encoder ->
let len = io_buffer_size - Def.dst_rem encoder in
if len > 0 then bigstring_output oc o 0 len
Is this a bug, or am I doing something wrong with the API?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.