juliaio / gzip.jl

A Julia interface for gzip functions in zlib

GZip.jl: A Julia interface for gzip functions in zlib

This module provides a wrapper for the gzip related functions of zlib, a free, general-purpose, legally unencumbered, lossless data-compression library. These functions allow the reading and writing of gzip files.

Usage

Typical usage would be something like

using GZip

# Write some text into a compressed .gz file
s = "gzip is part of zlib, a free, general-purpose, " *
    "legally unencumbered, lossless data-compression library"
fh = GZip.open("testfile.gz", "w")
write(fh, s)
close(fh)

# Read back the data
fh = GZip.open("testfile.gz")
s = readline(fh)
close(fh)

Notes

• This interface is only for gzipped files, not the streaming zlib compression interface. Internally, it depends on/uses the streaming interface, but the gzip related functions are higher level functions pertaining to gzip files only.
• GZipStream is an implementation of IO and can be used virtually anywhere IO is used.
• This implementation mimics the IOStream implementation, and should be a drop-in replacement for IOStream, with some caveats:
  • seekend and truncate are not available
  • readuntil is available, but is not very efficient. (But readline works fine.)

In addition to open, gzopen, and gzdopen, the following IO/IOStream functions are supported:

• close()
• flush()
• seek()
• skip()
• position()
• eof()
• read()
• readuntil()
• readline()
• write()
• peek()

Due to limitations in zlib, seekend and truncate are not available.

Old Documentation

Old documentation link: https://gzipjl.readthedocs.io/en/latest/

gzip.jl's Issues

[PkgEval] GZip may have a testing issue on Julia 0.4 (2014-10-20)

PackageEvaluator.jl is a script that runs nightly. It attempts to load all Julia packages and run their tests (if available) on both the stable version of Julia (0.3) and the nightly build of the unstable version (0.4). The results of this script are used to generate a package listing enhanced with testing results.

On Julia 0.4

• On 2014-10-19 the testing status was Tests pass.
• On 2014-10-20 the testing status changed to Tests fail, but package loads.

Tests pass. means that PackageEvaluator found the tests for your package, executed them, and they all passed.

Tests fail, but package loads. means that PackageEvaluator found the tests for your package, executed them, and they didn't pass. However, trying to load your package with using worked.

This issue was filed because your testing status became worse. No additional issues will be filed if your package remains in this state, and no issue will be filed if it improves. If you'd like to opt-out of these status-change messages, reply to this message saying you'd like to and @IainNZ will add an exception. If you'd like to discuss PackageEvaluator.jl please file an issue at the repository. For example, your package may be untestable on the test machine due to a dependency - an exception can be added.

Test log:

>>> 'Pkg.add("GZip")' log
INFO: Installing GZip v0.2.13
INFO: Package database updated

>>> 'using GZip' log
Julia Version 0.4.0-dev+1148
Commit 6290d34 (2014-10-20 05:03 UTC)
Platform Info:
  System: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

>>> test log
ERROR: opening file /home/idunning/julia04/usr/bin/../share/julia/helpdb.jl: No such file or directory
 in open at ./iostream.jl:117
 in open at ./iostream.jl:135
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in anonymous at no file:3
 in include at ./boot.jl:245
 in include_from_node1 at loading.jl:128
 in process_options at ./client.jl:293
 in _start at ./client.jl:362
 in _start_3B_3813 at /home/idunning/julia04/usr/bin/../lib/julia/sys.so
while loading /home/idunning/pkgtest/.julia/v0.4/GZip/test/gzip_test.jl, in expression starting on line 32
while loading /home/idunning/pkgtest/.julia/v0.4/GZip/runtests.jl, in expression starting on line 2


>>> end of log

support for `do` blocks in GZip `open` functions?

For Julia's Base.open, you can call it from a do block which automatically closes the file at the end and makes your code a little neater.

open("outfile", "w") do io
    write(io, data)
end

This behaves the same as python's recommended way of reading files.

with open('file', 'w') as f:
    f.write("content")

The source code for this is super simple.

function open(f::Function, args...)
    io = open(args...)
    try
        f(io)
    finally
        close(io)
    end
end

Is there any reason this functionality coulden't be implemented here? Or perhaps theres design reasons against it? If not, I'd be interested in making a pull request (so long as this is as easy as I think it looks).

dlopen warning message in Julia 0.4

There are some warning messages when I load GZip in Julia 0.4.

julia> using GZip
WARNING: dlopen is deprecated, use Libdl.dlopen instead.
 in depwarn at ./deprecated.jl:40
 in dlopen at deprecated.jl:10
 in include at ./boot.jl:250
 in include_from_node1 at ./loading.jl:129
 in include at ./boot.jl:250
 in include_from_node1 at ./loading.jl:129
 in reload_path at ./loading.jl:153
 in _require at ./loading.jl:68
 in require at ./loading.jl:51
WARNING: dlopen is deprecated, use Libdl.dlopen instead.
 in depwarn at ./deprecated.jl:40
 in dlopen at deprecated.jl:10
 in include at ./boot.jl:250
 in include_from_node1 at ./loading.jl:129
 in reload_path at ./loading.jl:153
 in _require at ./loading.jl:68
 in require at ./loading.jl:51

GZip in the devel METADATA checks for BitsKind

$ /usr/bin/julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.2.0+nightly20130428.2356fb8
 _/ |\__'_|_|_|\__'_|  |  -Linux-x86_64 (2013-04-28 17:58:13)
|__/                   |

julia> BitsKind
ERROR: BitsKind not defined

julia> Base.BitsKind
ERROR: BitsKind not defined

julia> using GZip

julia> gzf = gzopen("/tmp/foo.gz", "w")
GZipStream(<file /tmp/foo.gz>)

julia> println(gzf, 0)
ERROR: in write: BitsKind not defined
 in write at /home/bates/.julia/GZip/src/GZip.jl:443
 in show at show.jl:25
 in print at string.jl:3
 in print at string.jl:4
 in println at string.jl:5

Implement `peek`?

Would it be easy to implement peek(io) for compressed files?

readline() and chomp/keep

readline(s::GZipStream) does not handle the chomp keyword that is available in Base, readline(s::IO; chomp::Bool=true), and this causes some confusion.

Calling readline(s::GZipStream) thus gives lines with trailing newline, but readline(s::IO) does not. This is not reflected in the documentation when you type ?readline (or elsewhere).

(If you call readline(s::GZipStream, chomp=true/false) then the method in Base is invoked and the functionality is restored, but Base.readline() is significantly slower and not obvious from a user perspective.)

NB: chomp is replaced by keep in julia 0.7, but the issue remains.

Support for data streaming compression/decompression

I'd like to be able to use zlib's inflate/deflate functionality to compress/uncompress a stream of data in-memory. An example use-case is to be able to read compress data from or write compressed data to a file opened with mmap such that the entire contents of the file doesn't have to be loaded in-memory at once (useful for large files).

For reference: http://zlib.net/manual.html

x[i:] now deprecated

as of JuliaLang/julia@d5b5a1f:

WARNING: deprecated syntax "x[i:]" at /Users/jiahao/.julia/GZip/src/GZip.jl:460.
Use "x[i:end]" instead.

GZip shouldn't prompt the user to overwrite an existing file

If you GZip.open an existing file in "wb" mode, it prompts you at the console if you want to overwrite it, regardless of isinteractive().

shell> touch random.gz

julia> using GZip; GZip.open("random.gz", "wb")
random already exists -- do you wish to overwrite (y or n)?

I don't think we should do this. Instead we should follow the Linux Standard Base Specification, which says:

The mode argument is based on that of fopen()

i.e. it should follow C convention to silently truncate to zero length.

On Mac - ERROR: LoadError: could not load symbol "gzopen64": dlsym(0xfff147d636b0, gzopen64): symbol not found

Per here: JuliaML/MLDatasets.jl#86 it looks like this is failing on macOS Monterey. If I run test GZip in Julia 1.6 I get the same error as I got in the issue linked above:

ERROR: LoadError: could not load symbol "gzopen64":
dlsym(0xfff147d636b0, gzopen64): symbol not found
Stacktrace:
 [1] gzopen(fname::String, gzmode::String, gz_buf_size::Int64)
   @ GZip ~/.julia/packages/GZip/JNmGn/src/GZip.jl:249
 [2] gzopen(fname::String, gzmode::String)
   @ GZip ~/.julia/packages/GZip/JNmGn/src/GZip.jl:263
 [3] top-level scope
   @ ~/.julia/packages/GZip/JNmGn/test/runtests.jl:39
 [4] include(fname::String)
   @ Base.MainInclude ./client.jl:444
 [5] top-level scope
   @ none:6
in expression starting at /Users/logankilpatrick/.julia/packages/GZip/JNmGn/test/runtests.jl:30
ERROR: Package GZip errored during testing

Error loading zlib library

Hello, I'm a experienced R user but I'm new to julia. I'm using Windows 8 64-bit and julia-2b22323495. Trying to load GZip gives the following error:

julia> import GZip
ERROR: error compiling anonymous: could not load module zlib1: no error
in include_from_node1 at loading.jl:91 (repeats 2 times)
in reload_path at loading.jl:114
in require at loading.jl:48
at C:\Users\Luciano\AppData\Roaming\julia\packages\GZip\src\zlib_h.jl:8
at C:\Users\Luciano\AppData\Roaming\julia\packages\GZip\src\GZip.jl:74

I have downloaded the library and extracted in gzip\scr. Should it be somewhere else?

I must say that repeating import GZip a second time gives no error whatsoever.

Thanks

Add Appveyor support - Windows testing

As far as I can see, this must always fail? I am loath to try and solve this since I probably don't understand the original purpose.

If the code below fails (as it does on Windows at least), then some subsequent tests are not run - in the present master. But because of the try-catch statement, this may instead seem like a try at forcing test success? But that makes no sense?

runtests.jl:17
@windows_only gunzip="gunzip.exe"
@unix_only    gunzip="gunzip"
test_gunzip = true
 try
     run(`which $gunzip` |> DevNull)
 catch
     test_gunzip = false
 end

Out-of context test on Windows:

julia> run(pipeline(`which gunzip.exe`, DevNull))
ERROR: could not spawn `which gunzip.exe`: no such file or directory (ENOENT)
 in _jl_spawn at process.jl:262

A reasonable purpose here at the start might be to check if gunzip is present and and runnable on the system. But simply running it without command line arguments (like simply -h) would return an error:

julia> run(`gunzip.exe`)
(....)ERROR: failed process: Process(`gunzip.exe`, ProcessExited(1)) [1]
 in run at process.jl:531

Would this work on unix_only too?

@windows_only run(pipeline(`gunzip.exe -h`, DevNull))
@unix_only         run(pipeline(`gunzip.exe -h`, DevNull))

[PkgEval] GZip may have a testing issue on Julia 0.3 (2014-06-10)

PackageEvaluator.jl is a script that runs nightly. It attempts to load all Julia packages and run their tests (if available) on both the stable version of Julia (0.2) and the nightly build of the unstable version (0.3). The results of this script are used to generate a package listing enhanced with testing results.

On Julia 0.3

• On 2014-06-04 the testing status was Tests pass.
• On 2014-06-10 the testing status changed to Tests fail, but package loads.

Tests pass. means that PackageEvaluator found the tests for your package, executed them, and they all passed.

Tests fail, but package loads. means that PackageEvaluator found the tests for your package, executed them, and they didn't pass. However, trying to load your package with using worked.

This issue was filed because your testing status became worse. No additional issues will be filed if your package remains in this state, and no issue will be filed if it improves. If you'd like to opt-out of these status-change messages, reply to this message saying you'd like to and @IainNZ will add an exception. If you'd like to discuss PackageEvaluator.jl please file an issue at the repository. For example, your package may be untestable on the test machine due to a dependency - an exception can be added.

Test log:

INFO: Installing GZip v0.2.12
INFO: Package database updated
WARNING: @test_throws without an exception type is deprecated
     Use @test_throws EOFError write(gzfile,data)
 in backtrace at error.jl:30
 in include at boot.jl:244
 in anonymous at no file:3
 in include at boot.jl:244
 in include_from_node1 at loading.jl:128
WARNING: @test_throws without an exception type is deprecated
     Use @test_throws ErrorException gzopen(readall,test_compressed)
 in backtrace at error.jl:30
 in include at boot.jl:244
 in anonymous at no file:3
 in include at boot.jl:244
 in include_from_node1 at loading.jl:128
WARNING: @test_throws without an exception type is deprecated
     Use @test_throws ErrorException seek(gzfile,100)
 in backtrace at error.jl:30
 in include at boot.jl:244
 in anonymous at no file:3
 in include at boot.jl:244
 in include_from_node1 at loading.jl:128
WARNING: @test_throws without an exception type is deprecated
     Use @test_throws ErrorException truncate(gzfile,100)
 in backtrace at error.jl:30
 in include at boot.jl:244
 in anonymous at no file:3
 in include at boot.jl:244
 in include_from_node1 at loading.jl:128
WARNING: @test_throws without an exception type is deprecated
     Use @test_throws MethodError seekend(gzfile)
 in backtrace at error.jl:30
 in include at boot.jl:244
 in anonymous at no file:3
 in include at boot.jl:244
 in include_from_node1 at loading.jl:128
ERROR: ArgumentError("UTF32String data must be NULL-terminated")
 in UTF32String at utf32.jl:8
 in include at boot.jl:244
 in anonymous at no file:3
 in include at boot.jl:244
 in include_from_node1 at loading.jl:128
while loading /home/idunning/pkgtest/.julia/v0.3/GZip/test/gzip_test.jl, in expression starting on line 240
while loading /home/idunning/pkgtest/.julia/v0.3/GZip/runtests.jl, in expression starting on line 2
INFO: Package database updated

should implement `readbytes`

I hereby apologize for the readall function. I think GZip should implement readbytes instead, and inherit readall(::IO) that just wraps the result of readbytes in a string.

eachline() reports extra line in GZip file but not in unzipped file

I found an issue where eachline() was returning an extra empty line "" after the end of a gz file I was reading. The file ends in a single newline, and has 171 total lines. Reading the uncompressed file works fine, but as the output below shows reading from the GZip stream produces a spurious blank line.

This only happens for this file (thousands of other such files worked fine) and if I change the file more than just a character or two the bug goes away. Unfortunately this is medical data so I can't attach the file, but see the output below (using the current version of GZip):

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.0-rc4 (2014-08-15 04:01 UTC)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |  x86_64-redhat-linux
> using GZip
> open(f->length(readlines(f)), "/tmp/orig")
171
> GZip.open(fout->write(fout, open(readall, "/tmp/orig")), "/tmp/orig.gz", "w");
> GZip.open(f->length(readlines(f)), "/tmp/orig.gz")
172
> GZip.open(f->readlines(f), "/tmp/orig.gz")[end]
""
> open(fout->write(fout, GZip.open(readall, "/tmp/orig.gz")), "/tmp/orig2", "w");
> open(f->length(readlines(f)), "/tmp/orig2")
171

Empty strings make gzwrite throw an error

This works:

io = open("test.txt", "w")
write(io, "a")
write(io, "")
close(io)

This doesn't:

io = gzopen("test.txt.gz", "w9")
write(io, "a")
write(io, "") # throws GZError(0,"")
close(io)

Guessing the C code gzwrite calls doesn't like len = 0. I ran into this trying to save a DataFrame containing empty strings as a .csv.gz (print(io, df[i, j]) in printtable(io, df)).

Worth adding a check to line up with behavior of base IO?

`README.md` shows `import GZip` which probably should be `using GZip`

There is no need to import the package unless you plan to add methods, right?

Tests fail on Julia v0.7 RH Linux

   Testing GZip                                                                                                                                 
    Status `/tmp/tmpdQAYue/Manifest.toml`                                                                                                       
  [34da2185] Compat v1.0.1                                                                                                                      
  [92fee26a] GZip v0.4.0                                                                                                                        
  [2a0f44e3] Base64  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Base64`]                                                            
  [ade2ca70] Dates  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Dates`]                                                              
  [8bb1440f] DelimitedFiles  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/DelimitedFiles`]                                            
  [8ba89e20] Distributed  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Distributed`]                                                  
  [b77e0a4c] InteractiveUtils  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/InteractiveUtils`]                                        
  [76f85450] LibGit2  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/LibGit2`]                                                          
  [8f399da3] Libdl  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Libdl`]                                                              
  [37e2e46d] LinearAlgebra  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/LinearAlgebra`]                                              
  [56ddb016] Logging  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Logging`]                                                          
  [d6f4376e] Markdown  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Markdown`]                                                        
  [a63ad114] Mmap  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Mmap`]                                                                
  [44cfe95a] Pkg  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Pkg`]                                                                  
  [de0858da] Printf  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Printf`]                                                            
  [3fa0cd96] REPL  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/REPL`]                                                                
  [9a3f8284] Random  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Random`]                                                            
  [ea8e919c] SHA  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/SHA`]                                                                  
  [9e88b42a] Serialization  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Serialization`]                                              
  [1a1011a3] SharedArrays  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/SharedArrays`]                                                
  [6462fe0b] Sockets  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Sockets`]                                                          
  [2f01184e] SparseArrays  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/SparseArrays`]                                                
  [10745b16] Statistics  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Statistics`]                                                    
  [8dfed614] Test  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Test`]                                                                
  [cf7118a7] UUIDs  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/UUIDs`]                                                              
  [4ec0a83e] Unicode  [`~/local/julia-0.7/usr/bin/../share/julia/stdlib/v0.7/Unicode`]                                                          
Test Failed at /rhome/FNORO/.julia/packages/GZip/CdLVz/test/runtests.jl:143                                                                     
  Expression: file_size > sizeof(data)                                                                                                          
   Evaluated: 6970 > 6970                                                                                                                       
ERROR: LoadError: There was an error during testing                                                                                             
in expression starting at /rhome/FNORO/.julia/packages/GZip/CdLVz/test/runtests.jl:38                                                           
ERROR: Package GZip errored during testing                                                                                                      
Stacktrace:                                                                                                                                     
 [1] pkgerror(::String, ::Vararg{String,N} where N) at /rhome/FNORO/local/julia-0.7/usr/share/julia/stdlib/v0.7/Pkg/src/Types.jl:120            
 [2] macro expansion at ./logging.jl:313 [inlined]                                                                                              
 [3] #test#61(::Bool, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /rhome/FNORO/local/julia-0.7/usr/share/julia/stdlib/
v0.7/Pkg/src/Operations.jl:1283                                                                                                                 
 [4] #test at ./none:0 [inlined]                                                                                                                
 [5] #test#44(::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types

Unable to load on Windows 64-bit

My error message from this morning after a fresh Julia install (commit 72e1ffa) and a new init of my package directory. Trying to add DataFrames

ERROR: error compiling anonymous: could not load module libz: %1 is not a valid Win32 application.


 in include_from_node1 at loading.jl:91
at C:\Users\karbarcca\AppData\Roaming\julia\packages\GZip\src\zlib_h.jl:7
at C:\Users\karbarcca\AppData\Roaming\julia\packages\GZip\src\GZip.jl:73
at C:\Users\karbarcca\AppData\Roaming\julia\packages\DataFrames\src\RDA.jl:4
at C:\Users\karbarcca\AppData\Roaming\julia\packages\DataFrames\src\DataFrames.jl:251

WARNING: New definition

Hi,

It seems like that GZip is not working on my version of Julia: v"0.5.0-dev+3438"

`julia> Pkg.test("GZip")
INFO: Testing GZip
WARNING: New definition
write(GZip.GZipStream, Array{#T<:Any, N<:Any}) at /home/julien/.julia/v0.5/GZip/src/GZip.jl:456
is ambiguous with:
write(Base.IO, Array{UInt8, N<:Any}) at io.jl:154.
To fix, define
write(GZip.GZipStream, Array{UInt8, N<:Any})
before the new definition.
WARNING: readall is deprecated, use readstring instead.
[inlined code] from ./error.jl:26
in depwarn(::ASCIIString, ::Symbol) at ./deprecated.jl:64
[inlined code] from ./strings/io.jl:53
in readall(::IOStream) at ./deprecated.jl:30
in open(::Base.#readall, ::ASCIIString) at ./iostream.jl:114
[inlined code] from /home/julien/.julia/v0.5/GZip/test/runtests.jl:32
in anonymous at ./:4294967295
in include(::ASCIIString) at ./boot.jl:234
in include_from_node1(::UTF8String) at ./loading.jl:417
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
while loading /home/julien/.julia/v0.5/GZip/test/runtests.jl, in expression starting on line 27
Test Failed
Expression: write(gzfile,data) == length(data.data)
Evaluated: 440631 == 6449
ERROR: LoadError: There was an error during testing
in record(::Base.Test.FallbackTestSet, ::Base.Test.Fail) at ./test.jl:322
in do_test(::Base.Test.Returned, ::Expr) at ./test.jl:220
[inlined code] from ./test.jl:137
in anonymous at ./:4294967295
in include(::ASCIIString) at ./boot.jl:234
in include_from_node1(::UTF8String) at ./loading.jl:417
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
while loading /home/julien/.julia/v0.5/GZip/test/runtests.jl, in expression starting on line 27
================================[ ERROR: GZip ]=================================

failed process: Process(/usr/bin/julia -Cx86-64 -J/usr/lib/x86_64-linux-gnu/julia/sys.so --compile=yes --check-bounds=yes --code-coverage=none --color=yes /home/julien/.julia/v0.5/GZip/test/runtests.jl, ProcessExited(1)) [1]

ERROR: Base.Pkg.PkgError("GZip had test errors")
[inlined code] from ./strings/io.jl:33
in #test#55(::Bool, ::Any, ::Array{AbstractString,1}) at ./pkg/entry.jl:671
[inlined code] from ./int.jl:32
in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#test,Tuple{Array{AbstractString,1}}})() at ./pkg/dir.jl:31
in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#test,Tuple{Array{AbstractString,1}}}, ::ASCIIString) at ./file.jl:48
in #cd#1(::Array{Any,1}, ::Any, ::Any, ::Array{AbstractString,1}, ::Vararg{Array{AbstractString,1}}) at ./pkg/dir.jl:31
[inlined code] from ./boot.jl:304
in #test#3(::Bool, ::Any, ::ASCIIString, ::Vararg{ASCIIString}) at ./pkg.jl:228
in eval(::Module, ::Any) at ./boot.jl:237
`

'28 required packages:

• ApproxFun 0.1.0
• BlackBoxOptim 0.2.0
• Bokeh 0.2.0
• Cbc 0.2.0
• DataFrames 0.7.0
• Debug 0.1.6
• Distributions 0.8.10
• FactCheck 0.4.2
• FastGaussQuadrature 0.0.3
• ForwardDiff 0.1.6
• GLM 0.5.1
• Gadfly 0.4.2
• HDF5 0.5.8
• IJulia 1.1.9
• Ipopt 0.2.1
• JLD 0.5.9
• JuMP 0.12.2
• LaTeXStrings 0.1.6
• Lumberjack 2.0.1
• NLopt 0.3.1
• Optim 0.4.4
• PyCall 1.4.0
• PyPlot 2.1.1
• QuantEcon 0.4.2
• Roots 0.1.26
• Sobol 0.1.2
• StochasticSearch 0.2.2
• Winston 0.11.13
  65 additional packages:
• ApproXD 0.0.0- master (unregistered)
• ArrayViews 0.6.4
• AutoHashEquals 0.0.9
• BinDeps 0.3.21
• Blosc 0.1.4
• Cairo 0.2.31
• Calculus 0.1.14
• Clustering 0.5.0
• Codecs 0.1.5
• ColorTypes 0.2.2
• Colors 0.6.3
• Combinatorics 0.3.2
• CompEcon 0.0.0- master (unregistered)
• Compat 0.7.14
• Compose 0.4.2
• Conda 0.1.9
• Contour 0.1.0
• Copulas 0.0.0- master (unregistered)
• DSP 0.0.11
• DataArrays 0.2.20
• DataStructures 0.4.3
• Dates 0.4.4
• Distances 0.3.0
• Docile 0.5.23
• DualNumbers 0.2.2
• FileIO 0.0.4
• FixedPointNumbers 0.1.3
• FixedSizeArrays 0.1.0
• GZip 0.2.18
• Graphics 0.1.3
• Grid 0.4.0
• Hexagons 0.0.4
• IniFile 0.2.5
• Iterators 0.1.9
• JSON 0.5.0
• KernelDensity 0.1.2
• Lazy 0.10.1
• LightGraphs 0.5.2
• LightXML 0.2.1
• Loess 0.0.6
• MOpt 0.0.0- master (unregistered)
• MacroTools 0.3.0
• MathProgBase 0.4.3
• Measures 0.0.2
• Mustache 0.0.14
• NaNMath 0.2.1
• Nettle 0.2.3
• PDMats 0.4.1
• ParserCombinator 1.7.8
• Plots 0.5.4
• Polynomials 0.0.5
• Reexport 0.0.3
• Requires 0.2.2
• ReverseDiffSparse 0.5.5
• SHA 0.1.2
• Showoff 0.0.6
• Smolyak 0.0.0- master (unregistered)
• SortingAlgorithms 0.0.6
• StackTraces 0.1.1
• StatsBase 0.8.0
• StatsFuns 0.2.0
• Tk 0.3.7
• URIParser 0.1.3
• WoodburyMatrices 0.1.5
• ZMQ 0.3.1
  '

Any reason why?
Any help would be strongly appreciated.

Cheers,
Julien

GZip doesn't load on Mac OS X 10.9

tobias@ruediger ~> julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.0-prerelease+168 (2013-11-26 04:24 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit db2c6ef* (0 days old master)
|__/                   |  x86_64-apple-darwin13.0.0

julia> Pkg.add("Gadfly")
INFO: Installing Codecs v0.1.0
INFO: Installing Compose v0.1.19
INFO: Installing Datetime v0.1.2
INFO: Installing Distance v0.2.4
INFO: Installing Distributions v0.2.11
INFO: Installing Gadfly v0.1.25
INFO: Installing Iterators v0.1.1
INFO: Installing Loess v0.0.1
INFO: Installing NumericExtensions v0.2.18
INFO: REQUIRE updated.

julia> using Gadfly
Warning: New definition 
    |(NAtype,Any) at /Users/tobias/.julia/DataArrays/src/operators.jl:502
is ambiguous with: 
    |(Any,SynchronousStepCollection) at /Users/tobias/.julia/BinDeps/src/BinDeps.jl:286.
To fix, define 
    |(NAtype,SynchronousStepCollection)
before the new definition.
Warning: New definition 
    |(Any,NAtype) at /Users/tobias/.julia/DataArrays/src/operators.jl:502
is ambiguous with: 
    |(SynchronousStepCollection,Any) at /Users/tobias/.julia/BinDeps/src/BinDeps.jl:283.
To fix, define 
    |(SynchronousStepCollection,NAtype)
before the new definition.
ERROR: error compiling anonymous: could not load module libz: dlopen(libz.dylib, 1): image not found
 in include at boot.jl:238
 in include_from_node1 at loading.jl:114
 in include at boot.jl:238
 in include_from_node1 at loading.jl:114
 in reload_path at loading.jl:140
 in _require at loading.jl:58
 in require at loading.jl:46
 in include at boot.jl:238
 in include_from_node1 at loading.jl:114
 in reload_path at loading.jl:140
 in _require at loading.jl:58
 in require at loading.jl:46
 in include at boot.jl:238
 in include_from_node1 at loading.jl:114
 in reload_path at loading.jl:140
 in _require at loading.jl:58
 in require at loading.jl:43
while loading /Users/tobias/.julia/GZip/src/zlib_h.jl, in expression starting on line 8
while loading /Users/tobias/.julia/GZip/src/GZip.jl, in expression starting on line 74
while loading /Users/tobias/.julia/DataFrames/src/DataFrames.jl, in expression starting on line 18
while loading /Users/tobias/.julia/Gadfly/src/Gadfly.jl, in expression starting on line 4

or just GZip:

julia> using GZip
ERROR: error compiling anonymous: could not load module libz: dlopen(libz.dylib, 1): image not found
 in include at boot.jl:238
 in include_from_node1 at loading.jl:114
 in include at boot.jl:238
 in include_from_node1 at loading.jl:114
 in reload_path at loading.jl:140
 in _require at loading.jl:58
 in require at loading.jl:43
while loading /Users/tobias/.julia/GZip/src/zlib_h.jl, in expression starting on line 8
while loading /Users/tobias/.julia/GZip/src/GZip.jl, in expression starting on line 74

However libz is surely there:

tobias@ruediger ~> otool -L /usr/lib/libz.dylib
/usr/lib/libz.dylib:
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
tobias@ruediger ~> otool -D /usr/lib/libz.dylib
/usr/lib/libz.dylib:
/usr/lib/libz.1.dylib

Non-descriptive error if file does not exist

If the file being opened does not exist, the following error is thrown:

ERROR: GZError(-1, "gzopen failed")

I think it would be more useful to have a more informative error, such as a file not found message.

Tag a version

Please tag a version. The latest tagged version shows many deprecation warnings on julia 0.5. Master seems fine. With 0.5 RC0 out, we should get tagged versions to load without deprecation warnings.

Failing on v0.3 (using gc_enable)

After the latest update the following code fails because close calls gc_enable with a Bool argument, which I think is only supported on the master version of Julia right now. Not sure what the version support goals are for GZip, but I thought I should mention this for others who run into it.

f = GZip.open("/tmp/test", "w")
close(f)

Inconsistent error for file not found

Heya 👋🏽 thanks for making the lib available. This is more of a question, but it took some time in the afternoon.
Due to some relative path mistakes, I had some tests (that required the decompression of a gzip file passing locally), but failing on CI, which returned this message:

could not load library "libc"
  /usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header

which seems to be coming from this line

Digging deeper, it seems that there is a slight difference from Oses (below tests in ubuntu 22.04 in WSL2):

julia> unsafe_load(cglobal((:errno, :libc), Cint))
ERROR: could not load library "libc"
libc.so: cannot open shared object file: No such file or directory
Stacktrace:
 [1] top-level scope
   @ REPL[1]:1

julia> cglobal((:errno, "libc.so.6"), Cint)
Ptr{Int32} @0x00007f4cbe436038

A colleague using pop os and arch reported an error similar to the ELF header above.

Instead, shouldn't we use:

using Base.Libc
errno()

to report the error number in that block

`write(::GZipStream, ::UInt8)` has incorrect return value

Expected behavior: write should always return the number of bytes written.
Actual behavior: write returns the value written, but only if the value is a UInt8.
To reproduce:

using GZip

touch("test.gz")
GZip.open("test.gz", "w") do io
  write(io, 10)
end # Returns 8 as expected, since Int64 has 8 bytes.

GZip.open("test.gz", "w") do io
  write(io, 0xa)
end # Returns 10, but should return 1, since UInt8 has 1 byte.

Benchmark against (and package with?) other Zlib implementations

In #32 @jakebolewski suggested benchmarking GZip.jl against other zlib implementations such as the one by @jtkukunas and @cloudflare.

Would be interesting to do a comparison along the lines of this and samtool's, and possibly package these forks as other possible binary dependencies.

GZip doesn't load on Windows

Hi Kevin, I get the following error on Windows when loading DataFrames, which stems from GZip:

julia> using DataFrames
ERROR: error compiling anonymous: could not load module zlib1: %1 is not a valid Win32 application

 in include at boot.jl:238
 in include_from_node1 at loading.jl:96
 in include at boot.jl:238
 in include_from_node1 at loading.jl:96
 in reload_path at loading.jl:121
 in normpath at path.jl:100
 in fill! at array.jl:176WARNING:
backtraces on your platform are often misleading or partially incorrect

at C:\Users\theodore\AppData\Roaming\julia\packages\GZip\src\zlib_h.jl:8
at C:\Users\theodore\AppData\Roaming\julia\packages\GZip\src\GZip.jl:74
at C:\Users\theodore\AppData\Roaming\julia\packages\DataFrames\src\DataFrames.jl:16

I am aware this issue was raised before and was resolved, but it seems to appear again.

Pkg.test("GZip") fails on Windows

I'm encountering following error when I run Pkg.test("GZip")

julia> Pkg.test("GZip")
INFO: Testing GZip
ERROR: LoadError: unlink: resource busy or locked (EBUSY)
 in unlink at fs.jl:102
while loading C:\Users\Administrator\.julia\v0.4\GZip\test\runtests.jl, in expression starting on line 27
================================[ ERROR: GZip ]=================================


failed process: Process(`'C:\Users\Administrator\AppData\Local\Julia-0.4.3\bin\julia' --check-bounds=yes --code-coverage=none --color=yes 'C:\Users\Administrator\.julia\v0.4\GZip\test\runtests.jl'`, ProcessExited(1)) [1]

================================================================================

ERROR: GZip had test errors
 in error at error.jl:21

I encountered same error on 3 different Windows systems

• Windows 10 64 bit
• Windows 2008 R2 Service pack 1 (2)

All these 3 systems had Julia version 0.4.3

I tested this on following GZip versions

• v"0.2.18"
• v"0.2.18+" (Master)

I tried executing the runtests.jl manually, I encountered similar error

julia> include("runtests.jl")
ERROR: LoadError: unlink: resource busy or locked (EBUSY) in unlink at fs.jl:102
while loading C:\Users\julia\.julia\v0.4\GZip\test\runtests.jl, in expression starting on line 27

Fix ambiguity warning on master

Caused by JuliaLang/julia#14935

Cc: @vtjnash

Noted here:

julia> Pkg.test("GZip")
INFO: Testing GZip
WARNING: New definition
    write(GZip.GZipStream, Array{#T<:Any, N<:Any}) at /Users/kmsquire/.julia/v0.5/GZip/src/GZip.jl:456
is ambiguous with:
    write(Base.IO, Array{UInt8, N<:Any}) at io.jl:154.
To fix, define
    write(GZip.GZipStream, Array{UInt8, N<:Any})
before the new definition.

Version info:

julia> versioninfo()
Julia Version 0.5.0-dev+2440
Commit 2bb94d6 (2016-02-01 02:22 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Core(TM)2 Extreme CPU X7900  @ 2.80GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Core2)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1

Please tag a new version?

Could you please tag a new version of GZip so that we can get the fixes from a01a29b ? Thank you.

Fails when try to use the zlib on OSX 10.11.5 (64 bits)

I was using the library on OSX 10.11.5 and it failed when trowed to access the zlib library (64 bits and version 1.2.8). It throw the next error:

ERROR: ccall: could not find function gzopen64 in library libz
 in gzopen at /Users/eloy/.julia/v0.4/GZip/src/GZip.jl:249

To go through it, I've modified the file ~/.julia/v0.4/GZip/src/GZip.jl from the lines 222 to 225. To go from this:

222         const _gzopen = :gzopen64
223         const _gzseek = :gzseek64
224         const _gztell = :gztell64
225         const _gzoffset = :gzoffset64

to this:

222         const _gzopen = :gzopen
223         const _gzseek = :gzseek
224         const _gztell = :gztell
225         const _gzoffset = :gzoffset  

Thanks for the library and I hope this solution helps to someone else...

implement uvtype(::GZip.GZipStream)

This code runs ok:

julia> run(pipeline(open("input.csv", "r"), `cat`, open("output.csv", "w")))

but if I use GZip streams as input or output, it complains about uvtype method:

using GZip
run(pipeline(open("input.csv", "r"), `cat`, GZip.open("output.csv.gz", "w")))

LoadError: MethodError: no method matching uvtype(::GZip.GZipStream)
Closest candidates are:
  uvtype(!Matched::RawFD) at process.jl:158
  uvtype(!Matched::Base.DevNullStream) at process.jl:151
  uvtype(!Matched::Base.Filesystem.File) at filesystem.jl:70
  ...
while loading In[5], in expression starting on line 2

 in _jl_spawn(::String, ::Array{String,1}, ::Ptr{Void}, ::Base.Process, ::Base.Filesystem.File, ::GZip.GZipStream, ::RawFD) at ./process.jl:312
 in #412 at ./process.jl:478 [inlined]
 in setup_stdio(::Base.##412#413{Cmd,Ptr{Void},Base.Process}, ::Tuple{IOStream,GZip.GZipStream,RawFD}) at ./process.jl:466
 in #spawn#411(::Nullable{Base.ProcessChain}, ::Function, ::Cmd, ::Tuple{IOStream,GZip.GZipStream,RawFD}, ::Bool, ::Bool) at ./process.jl:477
 in (::Base.#kw##spawn)(::Array{Any,1}, ::Base.#spawn, ::Cmd, ::Tuple{IOStream,GZip.GZipStream,RawFD}, ::Bool, ::Bool) at ./<missing>:0
 in #spawn#408(::Nullable{Base.ProcessChain}, ::Function, ::Base.CmdRedirect, ::Tuple{RawFD,GZip.GZipStream,RawFD}, ::Bool, ::Bool) at ./process.jl:359
 in (::Base.#kw##spawn)(::Array{Any,1}, ::Base.#spawn, ::Base.CmdRedirect, ::Tuple{RawFD,GZip.GZipStream,RawFD}, ::Bool, ::Bool) at ./<missing>:0
 in #spawn#408(::Nullable{Base.ProcessChain}, ::Function, ::Base.CmdRedirect, ::Tuple{RawFD,RawFD,RawFD}, ::Bool, ::Bool) at ./process.jl:359
 in run(::Base.CmdRedirect) at ./process.jl:591

Is is possible to define this method?

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Error: could not load library libz

As reported by @juanklopper: JuliaPlots/Plots.jl#386 (comment)

julia ~/.julia/GZip/runtests.jl fails

$ ./julia -E 'Pkg.installed("GZip")'
v"0.2.10"
$ ./julia ~/.julia/GZip/runtests.jl 
ERROR: test failed: :((data==data3))
 in error at error.jl:21
 in default_handler at test.jl:19
 in do_test at test.jl:39
 in anonymous at no file:151
 in anonymous at no file:3
 in include_from_node1 at loading.jl:120
while loading /home/rick/.julia/GZip/test/gzip_test.jl, in expression starting on line 111
while loading /home/rick/.julia/GZip/runtests.jl, in expression starting on line 2
$ ./julia -e 'versioninfo()'
Julia Version 0.3.0-prerelease+1557
Commit 1a31530* (2014-02-13 05:06 UTC)
Platform Info:
  System: Linux (i686-redhat-linux)
  CPU: Genuine Intel(R) CPU           T2250  @ 1.73GHz
  WORD_SIZE: 32
  BLAS: libopenblas (DYNAMIC_ARCH NO_AFFINITY)
  LAPACK: libopenblas
  LIBM: libopenlibm

PackageEvaluator error

##### Current package: GZip
INFO: Installing GZip v0.2.10
INFO: REQUIRE updated.
ERROR: test failed: :((data==data3))
 in error at error.jl:21
 in default_handler at test.jl:19
 in do_test at test.jl:39
 in anonymous at no file:151
 in include at boot.jl:238
at /home/idunning/pkgtest/.julia/v0.2/GZip/test/gzip_test.jl:154
at /home/idunning/pkgtest/.julia/v0.2/GZip/runtests.jl:4
INFO: Removing GZip v0.2.10
INFO: REQUIRE updated.

64bit Arch Linux on Julia v0.2.1+2

zlib1 problem on windows server in Travis CI

Hi there,

I've been getting GZip errors related to zlib1 in a Julia 1.3 windows server Travis pipeline for a Julia package I'm developing. It's important to note that I do not have this problem in Julia 1.2 in the same Travis environment.

On the first try, I got the following error: (full output with stacktrace here)

ERROR: LoadError: LoadError: error compiling top-level scope: could not load library "zlib1"
The specified module could not be found.

I saw from this post that a potential solution is to download zlib1.dll and place it in the julia/lib folder. I tried that and had no success, so I tried placed the .dll in julia/bin instead. That caused a slightly different error: (full output with stacktrace here)

ERROR: LoadError: LoadError: error compiling top-level scope: could not load library "zlib1"
%1 is not a valid Win32 application.

I've tried quite a few things but have had no luck. I'm not sure what to try next, so any help would be greatly greatly appreciated.

Thanks!

GZip.IOBuffer type?

So I'm not super familiar with how the package is currently structured and am meaning to dive in more, but I wonder if someone could help clarify if this is a good idea or not.

I'm thinking of creating a GZip.IOBuffer type that would be a wrapper around a IOBuffer type, but that you could write to and whatever you write gets gzipped into the buffer. You could then do a takebuf_string to get the raw gzip data and send in an HTTP request, for example.

Is this amenable to how GZip compression works? Gzipping chunks at a time like this? I think the approach should be pretty simple to implement, but if there's anything I should watch out for or avoid, I'd love to hear it.