mirage / decompress

Pure OCaml implementation of Zlib.

Build Unikernels in OCaml

---

MirageOS is a library operating system that constructs secure, performant and resource-efficient unikernels.

About

MirageOS is a library operating system that constructs unikernels for secure, high-performance network applications across various cloud computing and mobile platforms. Developers can write code on a traditional OS such as Linux or macOS. They can then compile their code into a fully-standalone, specialised unikernel that runs under the Xen or KVM hypervisors and lightweight hypervisors like FreeBSD's BHyve, OpenBSD's VMM. These unikernels can deploy on public clouds, like Amazon's Elastic Compute Cloud and Google Compute Engine, or private deployments.

The most up-to-date documentation can be found at the homepage. The site is a self-hosted unikernel. Simpler skeleton applications are also available online. MirageOS unikernels repositories are also available here or there.

This repository

This repository contains the mirage command-line tool to create and deploy applications with MirageOS. This tool wraps the specialised configuration and build steps required to build MirageOS on all the supported targets.

Local install

You will need the following:

• a working OCaml compiler (4.08.0 or higher).
• the Opam source package manager (2.1.0 or higher).
• an x86_64 or armel Linux host to compile Xen kernels, or FreeBSD, OpenBSD or MacOS X for the solo5 and userlevel versions.

Then run:

$ opam install mirage
$ mirage --version

This should display at least version 4.0.0.

Using mirage

There are multiple stages to using mirage:

• write config.ml to describe the components of your applications;
• call mirage configure to generate the necessary code and metadata;
• optionally call make depends to install external dependencies and download Opam packages in the current dune workspace.
• call dune build to build a unikernel.

You can find documentation, walkthroughs and tutorials over on the MirageOS website. The install instructions are a good place to begin!

decompress's Issues

Infinite loop and eats all memory

I have this code

(* Data is a Bigarray.Array1 of chars *)
(* FIXME: For some reason it eats memory like crazy on one sample *)
let unzlib_raw ba =
    let open Decompress in
    Printf.printf "ZLib: Before unpack\n";
    let data = Util.ba_to_bytes ba in
    let datalen = Bytes.length data in
    Printf.printf "ZLib: datalen = 0x%x\n" datalen;
    Out_channel.flush Out_channel.stdout;
    let inputbuf = Bytes.create 0xFFFF in
    let outputbuf = Bytes.create 0xFFFF in
    let window = Window.create ~proof:B.proof_bytes in
    let pos = ref 0 in
    let res = Buffer.create datalen in
    Zlib_inflate.bytes inputbuf outputbuf
    (fun inputbuf ->
        let n = min 0xFFFF (datalen - !pos) in
        (* TODO: More efficient algorithm without copy *)
        Bytes.blit data !pos inputbuf 0 n;
        pos := !pos + n;
        n)
    (fun outputbuf len ->
        Printf.printf "ZLib: outfun len = 0x%x\n" len;
        Out_channel.flush Out_channel.stdout;
        Buffer.add_subbytes res outputbuf 0 len;
        0xFFFF)
    (Zlib_inflate.default window)
    |> function
        | Ok _ ->
                Printf.printf "ZLib: Unpack successful\n";
                Some (Buffer.contents res)
        | Error e ->
                Printf.printf "ZLib: Unpack failure\n";
                None

And I have a chunk of the data (it might be not a valid Zlib data, but it should still return a failure then, instead of perpetuating and eating all of memory until it gets OOM-killed. Attaching a ZIP archive of it (note it is not the ZIP causing problems, it is faulty_data.bin file inside)
faulty_data.zip

Library "decompress" not found with dune.2.5.1 and Multicore OCaml 4.10.0

I am trying to build decompress.1.1.0 using dune.2.5.1 for Multicore OCaml 4.10.0. It builds and installs fine, but, when I try to use it using:

(libraries checkseum.ocaml decompress)

I get the following error:

File "benchmarks/decompress/dune", line 4, characters 29-39:
4 |   (libraries checkseum.ocaml decompress))
                                 ^^^^^^^^^^
Error: Library "decompress" not found.

Source: https://github.com/ocaml-bench/sandmark/blob/master/benchmarks/decompress/dune
The above works fine with decompress.0.8.1 with dune-2.5.1 and Multicore OCaml 4.10.0 though! As mentioned in the README for issue #47, I then tried the following syntax:

(libraries (checkseum.ocaml decompress)))

But, I get the following error:

File "benchmarks/decompress/dune", line 4, characters 13-41:
4 |   (libraries (checkseum.ocaml decompress)))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Atom or quoted string expected

How can this be resolved?

Okasaki's queue and stdlib's queue

Currently, deflate algorithm use an Oakasaki's queue and it could be more interesting to find a data-structure which concatenation is fast with same semantic of Queue.

Then, in the blocklz algorithm, we use the stdlib's queue which is a double linked-list. From some benchmark in some other works (encore and faraday), the stdlib's is not the best choice in some contexts. So we need to figure about that (context & benchmark).

This issue is a starter to optimize (again) decompress. However, we already found the bottleneck - the adler-32 checksum.

status?

Can I use that library now or should I wait a little bit longer? :-)

Buffer overflow in De.Inf.Ns.inflate

I have a hit a buffer overflow in De.Inf.Ns.inflate. This has ultimately come from a call through ocaml-git decompressing a git pack, so it's not immediately clear to me what the input is at this point. I can capture it if you need it though.

The code in question is after line 1600 in lib/de.ml:

    let inflate lit dist d =
      try
        let rec inflate_loop () =
          __fill_bits d lit.Lookup.l
          ; let value =
              lit.Lookup.t.(d.hold land lit.Lookup.m) land Lookup.mask in
            let len = lit.Lookup.t.(d.hold land lit.Lookup.m) lsr 15 in
            d.hold <- d.hold lsr len
            ; d.bits <- d.bits - len
            ; if value < 256 then (
                unsafe_set_uint8 d.o d.o_pos value
                ; d.o_pos <- d.o_pos + 1
                ; inflate_loop ())
              else if value == 256 then raise_notrace End
              else
                let l = value - 257 in

The failure is an Invalid_argument from that unsafe_set_uint8. d.o has length 64k, and d.o_pos is 64k. My tracing shows that we've been around this loop filling up the buffer, and obviously this is the point that we've fallen off the end.

I don't have any clue what this code is doing, so I'm at a loss and would appreciate some assistance. I can see though that d.o_pos is being incremented around this loop, and there's no bounds checking that I can see, so I hope someone more knowledgeable than me could understand what's going on here.

Tagging @dinosaure in light of #100 :-)

LZO support

Outdated documentation

~proof should be replaced by ~witness in the following comment, right?

decompress/lib/decompress.mli

Line 186 in 3778626

(** [default ~proof ?wbits level] makes a new state [t]. [~proof] is an ['a

Inverted ctype for bench

Like xmlm to use Decompress in C++ benchmark.

+unit test

With alcotest. I love colors in my terminal.

Allow to specify wbits value in Inflate too

Like it is possible with "Zlib_deflate.default", like Python zlib allows it: https://docs.python.org/3/library/zlib.html

Unable to build benchmark locally

I was unable to build the benchmark in the way mentioned in the Benchmark section of the readme.

I was getting the following error when I tried to run it

 $ dune build bench/output.csv
Error: Don't know how to build bench/output.csv

My current opam switch is 4.12.1 and I do have dune installed locally

Any help on this would be great! 😃

LZMA support

Since it seems that ocaml-lzma is dead https://forge.ocamlcore.org/projects/ocaml-lzma/ and there are no libraries that provide a way to do that, it make sense to support it here too.

Exception raised when flushing buffers

Hello, I'm using decompress to perform Gz streaming compression of some ascii log. When killing my soft, I've a cleanup function that tries to gracefully empty up the internal buffers of decompress. Note that the gzip file produced was rather large at this stage (roughly 650 megabytes).

I noticed the following exception was raised:

Invalid_argument                                                                                                
    "Optint.to_int32: 13903751204 can not fit into a 32 bits integer")        

and indeed my gzip file is truncated at the end (checked using gzip -t), meaning that there was an error during this flushing down. Here's my flushing function (mostly inspired by the code for the pipe utility as you might notice):

      let rec flush_encoder oc encoder =
        let open Gz in
        match Def.encode encoder with
        | `Await encoder ->
            (* A length of 0 signals end of input (I think). *)
            flush_encoder oc (Def.src encoder i 0 0)
        | `Flush encoder ->
            let len = io_buffer_size - Def.dst_rem encoder in
            bigstring_output oc o 0 len ;
            flush_encoder oc (Def.dst encoder o 0 io_buffer_size)
        | `End encoder ->
            let len = io_buffer_size - Def.dst_rem encoder in
            if len > 0 then bigstring_output oc o 0 len

Is this a bug, or am I doing something wrong with the API?

Non full-stream interface

In some context such as Git, we are able to store a full zlib contents into one and unique buffer - when we manipulate small objects. Even if the stream API is interesting and useful for unknowable contents, we should have the opportunity to inflate a buffer and produce the output or fail such as:

val inflate : src:bigstring -> dst:bigstring -> (int * int, [> error ]) result
(** [inflate ~src ~dst] tries to inflate [src] and store the output into [dst]. It returns
    byte(s) consumed into [src] and byte(s) produced into [dst]. *)

Such implementation should be outside the current de.ml implementation and should just care about: we can inflate src or we will fail. This is what libdeflate does and, from this perspective, the constraint to save the world/state for a full-stream API disappears and we should be able to provide a fast inflation algorithm.

Mechanically, and due to the fact that ocaml-git/Git has such heuristic - it tries to inflate in one go -, I think we will improve performance a lot where most of Git objects are small enough to fit under a IO_BUFFER_SIZE buffer.

gzip

It's only about the header, so (I think), it's easy to implement this (@juanchanco did already this feature but in the old version of Decompress).

This issue is not for the next version of Decompress (1.0).

Fix for OCaml 4.08

Array1.map_file was removed in 4.08.0
See ocaml/opam-repository@0230f2f

Invalid_crc on a git blob.

With the following git pack:
http://tumbolandia.net/pack.dump

I get Exception: Decompress_inflate.Make(X).Invalid_crc., even though this seems to be perfectly fine for Zlib.

I used @yomimono's interface of Inflator available here: https://github.com/Engil/Canopy/blob/master/inflator.ml, and down here you can find a session of utop that reproduces the error.

───────────────────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────
                                                                                       │ Welcome to utop version 1.18.2 (using OCaml version 4.02.3)! │
                                                                                       └──────────────────────────────────────────────────────────────┘

Type #utop_help for help about using utop.

─( 19:18:02 )─< command 0 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # #require "irmin";;
─( 19:18:02 )─< command 1 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # #require "decompress";;
─( 19:18:11 )─< command 2 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # #require "git.unix";;
─( 19:18:17 )─< command 3 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # let load_file f =
  let ic = open_in f in
  let n = in_channel_length ic in
  let s = Bytes.create n in
  really_input ic s 0 n;
  close_in ic;
  (s);;
val load_file : bytes -> bytes = <fun>
─( 19:18:22 )─< command 4 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # let m =
     let b = load_file "/tmp/pack.dump" in
     Mstruct.of_string b;;
val m : Mstruct.t = <abstr>
─( 19:18:30 )─< command 5 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # #mod_use "Canopy/inflator.ml";;
module Inflator :
  sig
    module IDBytes :
      sig
        external length : bytes -> int = "%string_length"
        external get : bytes -> int -> char = "%string_safe_get"
        external set : bytes -> int -> char -> unit = "%string_safe_set"
        external create : int -> bytes = "caml_create_string"
        val make : int -> char -> bytes
        val init : int -> (int -> char) -> bytes
        val empty : bytes
        val copy : bytes -> bytes
        val of_string : bytes -> bytes
        val to_string : bytes -> bytes
        val sub : bytes -> int -> int -> bytes
        val sub_string : bytes -> int -> int -> bytes
        val extend : bytes -> int -> int -> bytes
        val fill : bytes -> int -> int -> char -> unit
        val blit : bytes -> int -> bytes -> int -> int -> unit
        val blit_string : bytes -> int -> bytes -> int -> int -> unit
        val cat : bytes -> bytes -> bytes
        val iter : (char -> unit) -> bytes -> unit
        val iteri : (int -> char -> unit) -> bytes -> unit
        val map : (char -> char) -> bytes -> bytes
        val mapi : (int -> char -> char) -> bytes -> bytes
        val trim : bytes -> bytes
        val escaped : bytes -> bytes
        val index : bytes -> char -> int
        val rindex : bytes -> char -> int
        val index_from : bytes -> int -> char -> int
        val rindex_from : bytes -> int -> char -> int
        val contains : bytes -> char -> bool
        val contains_from : bytes -> int -> char -> bool
        val rcontains_from : bytes -> int -> char -> bool
        val uppercase : bytes -> bytes
        val lowercase : bytes -> bytes
        val capitalize : bytes -> bytes
        val uncapitalize : bytes -> bytes
        type t = bytes
        val compare : t -> t -> int
        val unsafe_to_string : t -> t
        val unsafe_of_string : t -> t
        external unsafe_get : t -> int -> char = "%string_unsafe_get"
        external unsafe_set : t -> int -> char -> unit = "%string_unsafe_set"
        external unsafe_blit : t -> int -> t -> int -> int -> unit = "caml_blit_string" "noalloc"
        external unsafe_fill : t -> int -> int -> char -> unit = "caml_fill_string" "noalloc"
        val concat : t -> t list -> t
        val of_bytes : 'a -> 'a
        val to_bytes : 'a -> 'a
      end
    module XInflator :
      sig
        type t = Decompress_inflate.Make(IDBytes).t
        type src = [ `Channel of in_channel | `Manual of int -> bytes | `String of int * bytes ]
        type dst = bytes
        val make : [< src ] -> dst -> t
        val eval : t -> [ `Error | `Flush of int | `Ok of int ]
        val contents : t -> int
        val flush : t -> unit
      end
    module XDeflator :
      sig
        type t = Decompress_deflate.Make(IDBytes).t
        type src = [ `Channel of in_channel | `Manual of unit -> bytes * bool | `String of int * bytes ]
        type dst = bytes
        val make : ?window_bits:int -> [< src ] -> dst -> t
        val eval : t -> [ `Error | `Flush | `Ok ]
        val contents : t -> int
        val flush : t -> unit
      end
    val inflate : ?output_size:int -> Mstruct.t -> Mstruct.t option
    val deflate : ?level:'a -> Cstruct.t -> Cstruct.t
  end
─( 19:18:33 )─< command 6 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # Inflator.inflate (Mstruct.clone m);;
output_size: 0 wrote 416217 of 416217
Exception: Decompress_inflate.Make(X).Invalid_crc.
─( 19:18:46 )─< command 7 >─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # Git_unix.Zlib.inflate (Mstruct.clone m);;
- : Mstruct.t option = Some <abstr>```

ZSTD support

It is gaining popularity and it seems that OCaml bindings are abandoned: https://github.com/ygrek/ocaml-zstd

See format description https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md

And decoding testing tool https://github.com/facebook/zstd/tree/v1.3.4/tests#decodecorpus---tool-to-generate-zstandard-frames-for-decoder-testing

benchmark

Incorporating inflator.ml directly into decompress as high-level interface?

CC: @yomimono, @Engil, @verbosemode

Any plans for this? I just updated the inflator.ml to mirror Canopy's to make this work for the Syslog project: verbosemode/syslogd-mirage#9

Wondering if there is a better way we can do this other than copying files?

Include an even-higher level interface?

I had some trouble using this library since the higher-level interface is undocumented and still too-low-level for what I'm trying to do (decompress some data that I have in-memory).

I eventually came up with:

let inflate_string ?(buffer_size_bytes = 100000) data =
  let input = Bigstring.of_string data
  and buffer = Bigstring.create buffer_size_bytes
  and output = Queue.create () in
  let refill s = Bigstring.length s
  and flush s n = Queue.enqueue output (Bigstring.sub s ~pos:0 ~len:n) in
  Gz.Higher.uncompress ~refill ~flush input buffer
  |> Result.map ~f:(fun _ ->
         Queue.to_list output |> Bigstring.concat |> Bigstring.to_string)

Would it make sense to include something like this in the code, or at least give an example in the documentation?

Use jbuilder/dune

Now that we have a workaround for the "linking" hack: https://github.com/diml/variants-workaround it would be great to switch decompress to dune!

Explain `i_rem`

For reviewer, de.ml is a complex code base, I should explain some parts of it.

Optimization?

I have a dream, a dream with Decompress was 10 times faster than the implementation of C. Maybe it's not just a dream.

Flush block

We can implement (as zlib) a flush method (sync, partial or full) in Decompress. We need more test about that.

BGZF support

BGZF is a GZIP-based block compression format widely used in bioinformatics. BGZF uses extra fields as specified in RFC 1952 and is designed to be compatible with GZIP decompressors.

The Gz.Inf module decompressor fails to read data from a BGZF file. The MWE with its input file:

$ decompress --format gzip mwe.txt.gz
decompress: Unexpected end of input.

I wonder why the current GZIP decompressor implementation fails to read BGZF and whether it can be easily fixed.

The input file mwe.txt.gz was created in the following way using the bgzip tool:

$ seq 1 10 > mwe.txt               
$ bgzip mwe.txt  

Read 2 bytes per 2 bytes instead 1 and 1 byte.

A simple optimization should be to use get_int16_be(le?) here:

decompress/lib/de.ml

Lines 999 to 1004 in 366ff8d

	then ( hold := Nativeint.logor !hold Nativeint.(shift_left (of_int (unsafe_get_uint8 d.i !i_pos)) !bits)
	; bits := !bits + 8
	; incr i_pos
	; hold := Nativeint.logor !hold Nativeint.(shift_left (of_int (unsafe_get_uint8 d.i !i_pos)) !bits)
	; bits := !bits + 8
	; incr i_pos ) ;

The loop already ensures to have, at least 2 bytes available - so we can use an unsafe operation. However, we must prove that it's a real optimization - so we can use dinosaure/lzbench.

Know how many bytes decompress eat to inflate

Export RFC1951 interface

zlib wraps only the RFC1951 standard - so we already implemented it. It could be interesting to export only the RFC1951 decoder/encoder (in a smart way ...) and avoid the zlib header and the Adler-32 checksum at the end.

Bug with a specific file

When I try to serialize the PACK file for ocaml-git, I find a deep bug in the Inflate compute.

The file: https://paste.isomorphis.me/Aka

The Deflate computes without error the file (and zlib can inflate the result withtout error). So the problem is about the Inflate in decompress. It's too late in Phnom Penh, I will check what is going on tomorrow.

it would be nice to have very simple examples to show basic usage

The test codes have a different objective (and I already had a look at them).

Brotli support

Brotli is supported by most browsers as a content-encoding that compressed better than gzip and deflate. See:

• https://en.wikipedia.org/wiki/Brotli
• https://github.com/google/brotli

bin/pipe.ml isn't fixed for the new API

Still uses old functions and module names, some are already incompatible.

docker build?

It would be really nice if you could add a Dockerfile and automated build on Docker Hub so I could easily try this out without having to build it.

@avsm has a nice set of Ocaml base containers which make it easier.

Find the data end without actual unpacking

Formats like GZip do not define the end marker of the stream, and doing actual decompression for just finding out the size is overkill. Is it possible to do a "lighter" version of the walking algorithm to find the end of compressed data? And export this as an API.

decompress 1.3.0 tests fail to build due to bigarry pulling in dune

When I build the tests with NixOS, the following happens. Reproducible with 4.10, 4.11 and 4.12.

File "bindings/stubs/dune", line 15, characters 22-26:
15 |  (forbidden_libraries unix)
                           ^^^^
Error: Library "unix" was pulled in.
-> required by library "bigarray" in
   /nix/store/56bmf5x8l90znfdvigxbj92iwzdyrva9-ocaml-4.11.2/lib/ocaml
-> required by library "ctypes" in
   /nix/store/l91x7cq89449y7xcj1sq8645nkkmh6vh-ocaml4.11.2-ctypes-0.17.1/lib/ocaml/4.11.2/site-lib/ctypes
-> required by library "ctypes.stubs" in
   /nix/store/l91x7cq89449y7xcj1sq8645nkkmh6vh-ocaml4.11.2-ctypes-0.17.1/lib/ocaml/4.11.2/site-lib/ctypes
-> required by library "gen_decompress_bindings" in
   _build/default/bindings/stubs
        test alias test/runtest

Patching this line out makes the tests pass without a hitch. Is this safe to do?

Provide the stream API on the C side

Warnings and re

Outside the scope, decompress does not compile because we catch a warning as an error about deprecated re functions when we test it.

ocamlformat uses one column about static int array

/cc @gpetiot, I'm not sure about which option I should set to have a compact array declaration instead of:

decompress/lib/de.ml

Lines 222 to 480 in f5cdd9a

	let _length =
	[|
	0
	; 1
	; 2
	; 3
	; 4
	; 5
	; 6
	; 7
	; 8
	; 8
	; 9
	; 9
	; 10
	; 10
	; 11
	; 11
	; 12
	; 12
	; 12
	; 12
	; 13
	; 13
	; 13
	; 13
	; 14
	; 14
	; 14
	; 14
	; 15
	; 15
	; 15
	; 15
	; 16
	; 16
	; 16
	; 16
	; 16
	; 16
	; 16
	; 16
	; 17
	; 17
	; 17
	; 17
	; 17
	; 17
	; 17
	; 17
	; 18
	; 18
	; 18
	; 18
	; 18
	; 18
	; 18
	; 18
	; 19
	; 19
	; 19
	; 19
	; 19
	; 19
	; 19
	; 19
	; 20
	; 20
	; 20
	; 20
	; 20
	; 20
	; 20
	; 20
	; 20
	; 20
	; 20
	; 20
	; 20
	; 20
	; 20
	; 20
	; 21
	; 21
	; 21
	; 21
	; 21
	; 21
	; 21
	; 21
	; 21
	; 21
	; 21
	; 21
	; 21
	; 21
	; 21
	; 21
	; 22
	; 22
	; 22
	; 22
	; 22
	; 22
	; 22
	; 22
	; 22
	; 22
	; 22
	; 22
	; 22
	; 22
	; 22
	; 22
	; 23
	; 23
	; 23
	; 23
	; 23
	; 23
	; 23
	; 23
	; 23
	; 23
	; 23
	; 23
	; 23
	; 23
	; 23
	; 23
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 24
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 25
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 26
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 27
	; 28
	|]

Compression support

The current implementation of the GZIP interface exposes a compress function. However, the encoder does not handle any compression level (see Gz.Def.encoder interface documentation).

Is it planned to fully implement compression features through this library?

Build fails on 4.07.0+flambda

This isn't a blocker for me, just thought you might want to know!

build failure

$ opam install git
The following actions will be performed:
  ∗ install decompress 0.7    [required by git]
  ∗ install git        1.11.5
===== ∗ 2 =====
Do you want to continue? [Y/n] y

<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫 
[decompress.0.7] found in cache
[git.1.11.5] found in cache

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫 
[ERROR] The compilation of decompress failed at "/Users/ec/.opam/opam-init/hooks/opam-bin-cache.sh wrap
        0c7e7b06c487aeb762d2892b750061fb27621af6a0b292f1c9c4f901cf91b582 ocaml pkg/pkg.ml build --pinned false --tests
        false --with-cmdliner false".

#=== ERROR while compiling decompress.0.7 =====================================#
# context     2.0.0 | macos/x86_64 | ocaml-variants.4.07.0+flambda | https://opam.ocaml.org/#6d9b8fd1
# path        ~/Sync/Code/gitlabel-parser/_opam/.opam-switch/build/decompress.0.7
# command     ~/.opam/opam-init/hooks/opam-bin-cache.sh wrap 0c7e7b06c487aeb762d2892b750061fb27621af6a0b292f1c9c4f901cf91b582 ocaml pkg/pkg.ml build --pinned false --tests false --with-cmdliner false
# exit-code   1
# env-file    ~/.opam/log/decompress-69483-e7966f.env
# output-file ~/.opam/log/decompress-69483-e7966f.out
### output ###
# [...]
# ocamlfind ocamlopt unix.cmxa -I /Users/ec/.opam/default/lib/ocamlbuild /Users/ec/.opam/default/lib/ocamlbuild/ocamlbuildlib.cmxa -linkpkg myocamlbuild.ml /Users/ec/.opam/default/lib/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
# + ocamlfind ocamlopt unix.cmxa -I /Users/ec/.opam/default/lib/ocamlbuild /Users/ec/.opam/default/lib/ocamlbuild/ocamlbuildlib.cmxa -linkpkg myocamlbuild.ml /Users/ec/.opam/default/lib/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
# File "myocamlbuild.ml", line 1:
# Error: Files /Users/ec/.opam/default/lib/ocamlbuild/ocamlbuildlib.cmxa
#        and /Users/ec/Sync/Code/gitlabel-parser/_opam/lib/ocaml/stdlib.cmxa
#        make inconsistent assumptions over implementation Stdlib__sys
# Command exited with code 2.
# pkg.ml: [ERROR] cmd ['ocamlbuild' '-use-ocamlfind' '-classic-display' '-j' '4' '-tag' 'debug'
#      '-build-dir' '_build' 'opam' 'pkg/META' 'CHANGES.md' 'LICENSE.md'
#      'README.md' 'pkg/META' 'README.md' 'CHANGES.md' 'lib/decompress.a'
#      'lib/decompress.cmxs' 'lib/decompress.cmxa' 'lib/decompress.cma'
#      'lib/decompress.cmx' 'lib/decompress.cmi' 'lib/decompress.mli']: exited with 10

• OCaml: 4.07.0+flambda
• opam: 2.0.0
• Platform: macOS 10.14, Mojave

Index out of bounds in de.ml

Attaching is an archive of the input data buffer, that might be not a valid zlib data, but raise an exception.

Uncaught exception:

  (Invalid_argument "index out of bounds")

Raised by primitive operation at file "lib/de.ml", line 1010, characters 20-38
Called from file "lib/de.ml", line 1319, characters 27-37
Called from file "lib/gz.ml", line 450, characters 12-31
Called from file "lib/gz.ml" (inlined), line 553, characters 17-22

test.bin.zip

Gzip support

Gzip support seems to have gone missing in the 1.0 release.

Checkseum referenced from ~/.opam/default/lib/decompress/impl/decompress_impl.cmxa(Decompress_impl__Decompress_safe) - no implementations provided

After updating decompress project using decompress library fails to build with:

File "_none_", line 1:
Error: No implementations provided for the following modules:
         Checkseum referenced from /home/akochkov/.opam/default/lib/decompress/impl/decompress_impl.cmxa(Decompress_impl__Decompress_safe),
           /home/akochkov/.opam/default/lib/decompress/impl/decompress_impl.cmxa(Decompress_impl)

lto1: fatal error: bytecode stream in file ‘~/.opam/default/lib/checkseum/c/libcheckseum_c_stubs.a’ generated with LTO version 9.2 instead of the expected 9.4

I am not sure if this checkseum build problem. A few days I built decompress just fine, but today something got broken.

The system is Fedora 33 x86_64

[i] ℤ gcc --version                                                                                                                                                                                                               
gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1)
[i] ℤ ld --version                                                                                                                                                                                                                
GNU ld version 2.35-18.fc33
[i] ℤ ocamlc --version                                                                                                                                                                                                            
4.11.1
[i] ℤ opam --version                                                                                                                                                                                                              
2.0.6

Might be useful to also trigger then the checkseum rebuild somehow if/when decompress is updated.

[i] ℤ opam upgrade                                                                                                                                                                                                                11:51:41 
The following actions will be performed:
  ↻ recompile decompress 1.4.0 [upstream changes]
Do you want to continue? [Y/n] Y

<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[decompress.1.4.0] found in cache

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[ERROR] The compilation of decompress failed at "/home/akochkov/.opam/opam-init/hooks/sandbox.sh build dune build -p decompress -j 7".

#=== ERROR while compiling decompress.1.4.0 ===================================#
# context     2.0.6 | linux/x86_64 | ocaml-system.4.11.1 | https://opam.ocaml.org#3b10ec4f
# path        ~/.opam/default/.opam-switch/build/decompress.1.4.0
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -p decompress -j 7
# exit-code   1
# env-file    ~/.opam/log/decompress-960269-e7966f.env
# output-file ~/.opam/log/decompress-960269-e7966f.out
### output ###
#     ocamlopt bin/pipe.exe (exit 2)
# (cd _build/default && /usr/bin/ocamlopt.opt -w -40 -g -o bin/pipe.exe /home/akochkov/.opam/default/lib/bigarray-compat/bigarray_compat.cmxa /home/akochkov/.opam/default/lib/optint/optint.cmxa /home/akochkov/.opam/default/lib/checkseum/c/checkseum_c.cmxa -I /home/akochkov/.opam/default/lib/checkseum/c /usr/lib64/ocaml/unix.cmxa -I /usr/lib64/ocaml lib/de.cmxa lib/zl.cmxa lib/gz.cmxa /home/akoc[...]
# lto1: fatal error: bytecode stream in file ‘/home/akochkov/.opam/default/lib/checkseum/c/libcheckseum_c_stubs.a’ generated with LTO version 9.2 instead of the expected 9.4
# compilation terminated.
# lto-wrapper: fatal error: gcc returned 1 exit status
# compilation terminated.
# /usr/bin/ld: error: lto-wrapper failed
# collect2: error: ld returned 1 exit status
# File "caml_startup", line 1:
# Error: Error during linking (exit code 1)



<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build decompress 1.4.0
└─ 
╶─ No changes have been performed
~/data/ocaml took 6s 

Add foreign test vectors

Just so we know we agree with foreign implementations :)

Here are some:

• https://github.com/madler/zlib/blob/master/test/infcover.c
• https://github.com/imaya/zlib.js/tree/develop/test
• http://lkml.iu.edu/hypermail/linux/kernel/0808.3/2183.html

I wasn't able to find a set of official test vectors, but perhaps someone with stronger google-fu than me can be of assistance.