guzba / zippy Goto Github PK
View Code? Open in Web Editor NEWPure Nim implementation of deflate, zlib, gzip and zip.
License: MIT License
Pure Nim implementation of deflate, zlib, gzip and zip.
License: MIT License
Hello!
Is it in the plans to add support for tar.xz
files?
Thanks a lot and keep up the good work :)
Thanks for the library.
From what I can see, it supports only extracting all or adding all to a zip archive.
I have potentially huge archives and want to be able to add a file at a time (and write immediately flush to disk) when writing
and also to extract a single file from the archive, into memory (without writing the uncompressed to disk) when reading.
Maybe it could be something like:
let archive = ZipArchive("/path/to/my.zip", fmRead)
var buffer = newString(777)
# or
# var buffer = newSeq[int32](2048)
archive.extractInto("internal/zip/path.bin", buffer) # read into buffer[0].pointer
and for writing it could just support adding a single file or Stream at a time along with the compression level (if any).
Would this be something you'd consider supporting?
Since 8c43184 (product of git bisect
and tested on my local Mac) nimble test
fails at
zippy/tests/test_ziparchives.nim
Line 13 in 419ebae
I have seen memory usage as high as 4gb for certain tarballs, for example https://github.com/nim-lang/csources/archive/64e34778fa7e114b4afc753c7845dee250584167.tar.gz.
Hi, I was trying to do something like code below
Thanks for putting effort in this project.
proc addFile*(
archive: ZipArchive, content: string, path: string
) {.raises: [IOError, OSError, ZippyError].} =
## Adds a single file to the archive.
let (head, tail) = splitPath(path)
if head.len > 0 and head notin archive.contents:
archive.contents[(head & os.DirSep).toUnixPath()] =
ArchiveEntry(kind: ekDirectory)
archive.contents[path.toUnixPath()] = ArchiveEntry(
kind: ekFile,
contents: content,
lastModified: getTime(),
permissions: {},
)
I am trying to store files so that they keep the desire structure within the zipfile.
I see that there is addDir with signature: proc addDir(archive: ZipArchive, base, relative: string)
. I think it would be a good idea to expose it.
Something similar for addFile
would be appreciated. nim-lang/zip
has it.
When compiled for Arm 32-bit and run on a Raspberry Pi Zero, zippy throws the following RangeDefect:
Error: unhandled exception: value out of range: 2176188416 notin -2147483648 .. 2147483647 [RangeDefect]
This is because the proc parseFilePermissions()
is set up to read the platform-flexible signed int
, and externalFileAttr: uint32
is converted to an int
when parsing file permissions. It should instead be used as the uint32
it already is. We can fix this by changing the following two lines:
proc parseFilePermissions*(permissions: uint32): set[FilePermission] =
filePermissions: parseFilePermissions(externalFileAttr shr 16)
These changes negate the issue on the Raspberry Pi Zero, and hopefully should do the same on other 32-bit systems.
I can understand not supporting extraction of paths that are relative, but I think this is an edge case that isn't relative and also isn't supported:
Extracting paths containing `...` is not supported (nim-1.0.4/c_code/7_1/@m..@slib@spackages@[email protected])
This is from from nim-1.0.4-linux_x64.tar.xz.
I have similar checks in my library which it seems I was also not sure about (https://github.com/dom96/untar/blob/master/src/untar.nim#L149), but I think checking for just ".." is too broad.
I'm really glad for this library in our ZIP-utils.
We have started to use the library to create ZIP archives, which works fine until the ZIP files are larger than the available memory. In those cases we are getting the "out of memory".
Are there any hacks to overcome this? Procedures for using a stream, adding individual files instead of directories, or is it there no solution to this yet?
I've found that usiing Zippy to uncompress archive & tarball doesn't preserve symlink (which breaks some libraries package as archive)
compilation command:
nimble js -o:public/main.js src/main.nim
nimble: v0.11.0
nim: v1.0.6
output:
/home/codespace/.nimble/pkgs/zippy-0.5.5/zippy/common.nim(137, 13) Error: type mismatch: got <int literal(16), uint8>
but expected one of:
proc `-`(x: int8): int8
first type mismatch at position: 2
extra argument given
proc `-`(x: int16): int16
first type mismatch at position: 2
extra argument given
proc `-`(x, y: float32): float32
first type mismatch at position: 2
required type for y: float32
but expression 'length' is of type: uint8
proc `-`(x: float32): float32
first type mismatch at position: 2
extra argument given
proc `-`(x, y: int32): int32
first type mismatch at position: 2
required type for y: int32
but expression 'length' is of type: uint8
proc `-`(x: float): float
first type mismatch at position: 2
extra argument given
proc `-`(x: int32): int32
first type mismatch at position: 2
extra argument given
proc `-`(x, y: int8): int8
first type mismatch at position: 2
required type for y: int8
but expression 'length' is of type: uint8
proc `-`[T: SomeUnsignedInt](x, y: T): T
first type mismatch at position: 2
required type for y: T: SomeUnsignedInt
but expression 'length' is of type: uint8
proc `-`(x, y: int16): int16
first type mismatch at position: 2
required type for y: int16
but expression 'length' is of type: uint8
proc `-`(x: int): int
first type mismatch at position: 2
extra argument given
proc `-`(x, y: float): float
first type mismatch at position: 2
required type for y: float
but expression 'length' is of type: uint8
proc `-`(x: int64): int64
first type mismatch at position: 2
extra argument given
proc `-`(x, y: int64): int64
first type mismatch at position: 2
required type for y: int64
but expression 'length' is of type: uint8
proc `-`(x, y: int): int
first type mismatch at position: 2
required type for y: int
but expression 'length' is of type: uint8
1 other mismatching symbols have been suppressed; compile with --showAllMismatches:on to see them
Heya,
First off, thanks a ton for this library.
I'm not exactly sure what I'm doing wrong here, I have the following code:
import zippy
import zippy/ziparchives
import strformat
import os
const pyzip = slurp"../rsrc/python-3.9.1-embed-amd64.zip"
const tmpdir: string = getEnv("LOCALAPPDATA") / "Temp"
proc extractPython(): bool =
var z = uncompress(pyzip)
extractAll(z, tmpdir)
return true
proc main(): void =
echo fmt"[*] tmpdir: {tmpdir}"
if extractPython():
echo fmt"[*] Python extracted"
when isMainModule:
main()
I'm cross compiling to Windows from MacOS using Mingw. The above code produces the following error when run:
Does zippy not support decompressing zip files this way?
Thanks
Hey!
First time finally using your library. Thanks for your work first of all. :)
While trying to see how it performs on some of my larger .tar.gz
archives, I noticed that for one of the larger ones extraction via extractAll
fails with a size verification error. The issue seems to be that the archive is larger than int32.high
, due to the size check here:
https://github.com/guzba/zippy/blob/master/src/zippy/gzip.nim#L76
Given that isize
is already a uint32
why do we mod
the length of dst
by (1 shl 31)
and not allow the full uint32
range?
My archive has 476,932 files with a total size of 3,497,146,858 bytes (according to du . -b
in the extracted directory).
I also wondered what would happen if we fully exceed the size of uint32
, but with a quick try I wasn't able to build a .tar.gz
archive that actually enters the uncompressGzip
proc. Instead it seems to go here https://github.com/guzba/zippy/blob/master/src/zippy/tarballs.nim#L51 instead (I didn't actually check that, I only noticed that uncompressedGzip
wasn't being run for my ~8 GB deflated archive).
Code to reproduce is pretty much just:
import zippy/tarballs
const tmp = "/tmp/foobar"
extractAll("/path/to/file.tar.gz", tmp)
I didn't open a PR, because I assume the line mentioned above is the way it is for a good reason. If it helps I can also send you the archive via google drive or w/e.
Thanks!
Would be good
This may be an issue with me being new to nim, but if I want to keep a reference to a ZipArchiveReader
it's not exposed:
import zippy
type
Null0Game* = object
files*: ZipArchiveReader
proc newNull0Game*(filename: string): Null0Game =
## Create a new Game instance
var game: Null0Game
game.files = openZipArchive(filename)
return game
I get undeclared identifier: 'ZipArchiveReader'
. I think it just needs a few *
to allow type-reference.
I am happy to PR for this, if this is the right way.
Extracting with tar
:
$ tar --extract --file=./pcre-8.45.tar.gz
$ stat -c%a ./pcre-8.45/configure
755
Extracting with zippy
:
import pkg/zippy/tarballs
extractAll(tarPath = "./pcre-8.45.tar.gz", dest = "./pcre")
Result:
$ stat -c%a ./pcre/pcre-8.45/configure
75
Tarball: https://megalink.dl.sourceforge.net/project/pcre/pcre/8.45/pcre-8.45.tar.gz
This is not a call to arms, just something for me to remember that I need to look more into this. I am able to use 32-bit RISC-V just fine with zippy, but the signed shifts cause problems on 64-bit.
So what gives? Well, on RISC-V the arithmetic shifts (SRAI instructions primarily) will drag the sign bit with them down. So any time you have a signed value it will still be signed afterwards.
Example (just imagine that these are 64-bit signed integers instead of 16-bit):
0x8000 SRAI 8 == 0xFF80
0x8000 SRAI 12 == 0xFFF8
0x8000 SRAI 15 == 0xFFFF
Exactly which lines of code cause SRAI to be emitted is not known to me, but I believe it wouldn't be emitted for unsigned shifts.
As a pure Nim implementation, are there any plans to make this available for the JS backend?
I tried to compress the folder in which there were files with the vin code, everything worked well. Then I was wondering how to go inside folders inside archives and I created the following hierarchy:
code:
createTarball("../utils/testFolder", "examples.tar.gz")
var tarbol = Tarball()
tarbol.open("examples.tar.gz")
echo tarbol.contents
And I got this:
The wall of invalid data, but there are some correct things, For example I see key: "testFolder/sas"
examples.tar.gz
OS: Arch, nim: 1.4.8
Before 8c43184, tests/test_ziparchives.nim
was writting files in system temporary directories. This seems to no longer be the case.
This is breaking nixpkgs package tests: NixOS/nixpkgs#176342
error: builder for '/nix/store/y81y71annic3isgqg417w2sbd5ckg430-zippy-0.9.9.drv' failed with exit code 1;
last 10 log lines:
> inflating: tmp/zip/gold/nim-1.6.2/tools/vccexe/vccenv.nim
> inflating: tmp/zip/gold/nim-1.6.2/tools/vccexe/vccexe.nim
> inflating: tmp/zip/gold/nim-1.6.2/tools/vccexe/vccvswhere.nim
> inflating: tmp/zip/gold/nim-1.6.2/tools/vccexe/vcvarsall.nim
> inflating: tmp/zip/gold/nim-1.6.2/tools/website.nimf
> inflating: tmp/zip/gold/nim-1.6.2/tools/winrelease.nim
> fatal.nim:53 sysFatal
> Error: unhandled exception: test_ziparchives.nim:28:16 `fileExists(zippyPath)` [AssertionDefect]
> Error: execution of an external program failed: '/build/nimcache/test_ziparchives_49AB4550FB1DC32554F842F393B0A633A4FA67C8 '
> check phase failed
Is there a reason that this test is no longer using getTempDir
?
Best regards,
In the in-progress reworking for zip archives, one can no longer read from an arbitrary stream.
Also, while the move to memory mapped files is a great option, there can be cases where a process isn't going to have any write-access to the file system (which would include the file used for the memory map).
It would be preferable to be flexible here, but have good defaults. For instance, I'd love to have you reading from a stream I hand you, and only deflate one file at a time as I ask for them... and I would even like the option to hand you an output stream. That'd give me options to save on memory consumption as well.
when we use nimble install zippy
, the message says Downloading https://github.com/guzba/zippy using git
. the directory tests
takes up 35.1M, so it takes a long time to clone; ok to be honest, I have a bad internet supplier, for most of the time I can't finish git clone
. I have to download the zipped source, which will be success sometimes, then install by hand. As a result, it is pain for me to install nim's library.
So, I propose to seperate test and examples from zippy lib, as a result we can get a rapid installation of the lib.
thanks
zip file from:
https://github.com/google/material-design-icons/archive/refs/heads/master.zip
import zippy/ziparchives, zip/zipfiles
# -----------------------------------
let filename = "material-design-icons-master.zip"
let z = openZipArchive(filename)
var ix : int64
echo "zippy"
for f in z.walkFiles:
inc(ix)
echo ix," entries"
# -----------------------------------
ix = 0
var z1: zipfiles.ZipArchive
discard z1.open(filename)
echo "zip"
for f in z1.walkFiles:
inc(ix)
echo ix," entries"
results
zippy
48629 entries
zip
952432 entries
environment
nim 1.6.2 stable
Linux Mint 64
I'd love to see zippy support bzip2 extraction, there doesn't seem to be a working bz2 solution for Nim at the moment
Zip files created with zippy (at least on Windows 64) do not preserve the date/time of the original files. Instead, everything is dated 1980-00-00 00:00.
Another nim package, zip/zipfles, does preserve time/dates but it doesn't work on Windows 64. I'd really like to use zippy. Is there a way?
Since zip
and this library doesn't support reusing the same z_stream context for multiple messages.
In my own discord library, I would like to implement some transport compression which neither zip or zippy have this functionality of creating a zlib inflation context to run chunks through. (https://discord.com/developers/docs/topics/gateway#transport-compression-transport-compression-example)
When using zippy to extract zip files, I've noticed it mess up file permissions on Linux. Not sure what's causing it, but here's an example of it happening.
import zippy/ziparchives
extractAll("example.zip", "example")
results in
.-----x--- overflow1.png ... overflow10.png
.rw-r--r-- overflowneg1.png ... overflowneg10.png
instead of
.rw-r--r-- overflow1.png ... overflow10.png
.rw-r--r-- overflowneg1.png ... overflowneg10.png
https://github.com/bung87/finder/tree/zippy
I previously use std zip lib which works fine and test passed, when switch to zippy I got Unhandled exception: Unsupported zip archive [ZippyError]
the zip file compression through mac context menu, am on
platform: macOS
nim version: 1.4.4
Add support for Comment string in ZIP files (like zip -z "comment"
in Linux zip command).
Hi there,
thanks for this library!
At the moment I would be particularly interested in reading "gzipped" files but I didn't find the corresponding documentation. In particular, I need to read large files that would not fit in memory, either as streams or line by line.
Could you please help me with this?
Thank you very much
Andrea
It would be cool to be able to include/ignore files in createZipArchive
/addDir
with an optional glob-pattern:
Like this would include all top-level files in myfiles/
, but exclude .DS_Store
files:
createZipArchive("myfiles/*", "file.zip", ".DS_Store")
Or include all files in current directory recursively, except zip files:
createZipArchive("**/*", "file.zip", "*.zip")
If there is interest, I can PR for it.
I would like to be able to add symlinks to my tarball
It's a simple program that compresses and decompresses a string using gzip, it crashes on 32-bit machines (tested on Arch Linux 32 with Nim 1.4.2):
import zippy
let data = compress("Just for testing", BestCompression, dfGzip)
echo uncompress(data)
Crash log:
$ ./zippy_test
/home/victor/.nimble/pkgs/zippy-0.5.1/zippy.nim(84) zippy_test
/home/victor/.nimble/pkgs/zippy-0.5.1/zippy.nim(38) compress
/home/victor/.nimble/pkgs/zippy-0.5.1/zippy/deflate.nim(223) deflate
/home/victor/.nimble/pkgs/zippy-0.5.1/zippy/lz77.nim(62) lz77Encode
/usr/lib/nim/system/fatal.nim(49) sysFatal
Error: unhandled exception: value out of range: 2323877985 notin -2147483648 .. 2147483647 [RangeDefect]
On Windows, unzipping to a partition other than C:/
fails.
When unzipping, you creating a temporary folder & are writing archive contents to that temporary folder, then moving contents to the final folder. However, Nim currently has a bug where moveDir
operations across partitions fail.
I would suggest either:
moveDir(tempDir, dest)
with a copyDir
and removeDir
operation/home/hortinstein/.cache/nim/cross_compile_d/@m..@[email protected]@[email protected]@szippy@sadler32_simd.nim.c:9:10:
fatal error: arm_neon.h: No such file or directory
9 | #include "arm_neon.h"
I am getting the following issue when I try to compile, here is a reproduction:
https://github.com/hortinstein/cross_compile
nimble buildmobile
Code:
import pkg/zippy/tarballs
extractAll(tarPath = "./libressl-3.4.2.tar.gz", dest = "./libressl")
Log:
/home/snwmds/test_zippy.nim(3) test_zippy
/home/snwmds/.nimble/pkgs/zippy-0.9.5/zippy/tarballs.nim(63) extractAll
/home/snwmds/.nimble/pkgs/zippy-0.9.5/zippy/tarballs.nim(14) parseTarOctInt
[[reraised from:
/home/snwmds/test_zippy.nim(3) test_zippy
/home/snwmds/.nimble/pkgs/zippy-0.9.5/zippy/tarballs.nim(129) extractAll
]]
Error: unhandled exception: /home/snwmds/test_zippy.nim(3) test_zippy
/home/snwmds/.nimble/pkgs/zippy-0.9.5/zippy/tarballs.nim(63) extractAll
/home/snwmds/.nimble/pkgs/zippy-0.9.5/zippy/tarballs.nim(12) parseTarOctInt
/home/snwmds/.choosenim/toolchains/nim-1.6.2/lib/pure/strutils.nim(1143) parseOctInt
invalid oct integer: 100664 [ZippyError]
Tarball: https://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-3.4.2.tar.gz
It would be nice to have a procedure that just returns the binary of a zipfile instead of writing to disk.
thanks for this excellent library.
I'm a Windows user. When I clone your repo without any special treatment an then run the test using
nim c -r tests/test
, it will fail when comparing decompression result of rfctest2.gold
.
IIRC this can be solved using .gitattributes
file to prevent specific files undergo automatic EOL conversion.
In also see in your CI matrix, there is no windows build.
That's why you are not aware of this subtle difference related to git.
The error more specifically is this one:
tests/test_ziparchives_read.nim(46, 5) `entry == entries[numEntries]`
I don't understand why zip file catenation would depend upon string hasher order, but it is one of a couple things holding up activating nimPreviewHashFarm
by default. So, if you could look into it soon, that would be awesome.
Currently tarballs cannot have more than 100 characters. Resulting in an error.
export LC_ALL=C nimble test
should reproduce the failure in most environments.
More specifically, the test code tries to open "/tmp/zippy/Nim-1.6.6/tests/misc/#U00e5#U00e4#U00f6.nim"
(that literal string with '#' 'U' and so on in it). I did not track where within the bowels of test/nimble/zippy string handling the utf8 gets converted to that expanded representation in this mode, though.
control using : unzip
UnZip 6.00 of 20 April 2009, by Info-ZIP. Maintained by C. Spieler. Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.
zippy: 0.5.7
experienced with this zip file: https://impbox.net/nico/android.zip (3MB)
It appears the extra data may be coming from the file names, running zippy over the same file multiple times gives different results in the extra data.
using extractAll("android.zip", "zippy/android")
diff -ur unzip/android/ zippy/android/
diff -ur unzip/android/ zippy/android/
diff -ur unzip/android/app/jni/SDL/src/hidapi/android/jni/Application.mk zippy/android/app/jni/SDL/src/hidapi/android/jni/Application.mk
--- unzip/android/app/jni/SDL/src/hidapi/android/jni/Application.mk 2020-07-17 15:10:36.000000000 +1000
+++ zippy/android/app/jni/SDL/src/hidapi/android/jni/Application.mk 2021-05-08 09:06:37.330885000 +1000
@@ -1,2 +1,3 @@
APP_STL := gnustl_static
APP_ABI := armeabi-v7a
+▒riptenaudio.h
\ No newline at end of file
diff -ur unzip/android/app/jni/SDL/src/hidapi/bootstrap zippy/android/app/jni/SDL/src/hidapi/bootstrap
--- unzip/android/app/jni/SDL/src/hidapi/bootstrap 2020-07-17 15:10:36.000000000 +1000
+++ zippy/android/app/jni/SDL/src/hidapi/bootstrap 2021-05-08 09:06:37.333887400 +1000
@@ -1,2 +1,4 @@
#!/bin/sh -x
autoreconf --install --verbose --force
+e, 1);
+ }
\ No newline at end of file
diff -ur unzip/android/app/jni/SDL/src/hidapi/libusb/hidusb.cpp zippy/android/app/jni/SDL/src/hidapi/libusb/hidusb.cpp
--- unzip/android/app/jni/SDL/src/hidapi/libusb/hidusb.cpp 2020-07-17 15:10:36.000000000 +1000
+++ zippy/android/app/jni/SDL/src/hidapi/libusb/hidusb.cpp 2021-05-08 09:06:37.357888200 +1000
@@ -1,3 +1,4 @@
#define NAMESPACE HIDUSB
#include "hid.c"
+c
\ No newline at end of file
diff -ur unzip/android/app/jni/SDL/src/hidapi/linux/hidraw.cpp zippy/android/app/jni/SDL/src/hidapi/linux/hidraw.cpp
--- unzip/android/app/jni/SDL/src/hidapi/linux/hidraw.cpp 2020-07-17 15:10:36.000000000 +1000
+++ zippy/android/app/jni/SDL/src/hidapi/linux/hidraw.cpp 2021-05-08 09:06:37.371886400 +1000
@@ -1,3 +1,4 @@
#define NAMESPACE HIDRAW
#include "hid.c"
+e.h
\ No newline at end of file
Only in unzip/android/app/src/main: AndroidManifest.xml
Binary files unzip/android/app/src/main/res/mipmap-hdpi/ic_launcher.png and zippy/android/app/src/main/res/mipmap-hdpi/ic_launcher.png differ
Binary files unzip/android/app/src/main/res/mipmap-xhdpi/ic_launcher.png and zippy/android/app/src/main/res/mipmap-xhdpi/ic_launcher.png differ
Binary files unzip/android/app/src/main/res/mipmap-xxhdpi/ic_launcher.png and zippy/android/app/src/main/res/mipmap-xxhdpi/ic_launcher.png differ
Only in unzip/android/app/src/main/res/values: strings.xml
diff -ur unzip/android/settings.gradle zippy/android/settings.gradle
--- unzip/android/settings.gradle 2020-07-17 15:10:40.000000000 +1000
+++ zippy/android/settings.gradle 2021-05-08 09:06:38.497886400 +1000
@@ -1 +1,2 @@
include ':app'
+
\ No newline at end of file
It would be a nice feature to be able to choose a destination folder that already exists. For example:
someDir/
- myZip.zip
And extract it like this:
someDir/
- myZip.zip
- the content of myZip.zip
The reason for this is the inconvenience of having to nest the destination in a new folder like this (plus the zip file may itself contain a folder as the top element):
someDir/
- myZip.zip
- myUnzippedFolder/
- the content of myZip.zip
If you don't want to deal with overwriting files that already exist, you could perhaps demand from the user that they clean all old copies away before extracting. And otherwise you raise an exception if you stumble upon a clash.
What do you think? 😄
I want to extract a Tarball located at tarPath
to tmpDir
(not existing sub-directory). However, nothing gets extracted.
When I extract this Tarball with PeaZip, everything gets extracted perfectly and works fine. However, when I do it with this procedure
proc extractAll(tarPath, dest: string) {.raises: [IOError, ZippyError, OSError], tags: [
ReadIOEffect, ReadDirEffect, ReadEnvEffect, WriteDirEffect, WriteIOEffect].}
the files never get extracted.
There is no exception, no message, nothing. No idea what is going on there.
The Tarball is named properly, suffixed with a .tar
and works perfectly fine.
After debugging tarball.nim
I found out that inside the extractAll
procedure, the loop after
Line 200 in bd02047
open
, this procedure seems to work fine (but perhaps is not adding the files properly to the tarball
) and finds all filenames from the tarball.
As expected, the provided Tarball()
does not have contents, so nothing is extracted, as nothing is provided by open()
:
ref 0000000004913fb0 --> [contents = [data = [],
counter = 0,
first = -1,
last = -1]]
Getting the above when running echo tarball.repr
here:
Line 200 in bd02047
Through further debugging open()
my suspicion was confirmed: open()
never fills the tarball.contents
; they always remain empty.
Line 109 in bd02047
The above line ensures that tarball.contents
are only filled, when the typeFlag
either equals 0
or 5
.
The problem is: typeFlag
is always empty!
Line 100 in bd02047
The above line should ensure, that a valid fileNamePrefix
is given, if the header is specifying a ustar
archive.
The issue is, that fileNamePrefix
always remains empty. because neither header[345 ..< 500].trim()
nor header[345 ..< 500]
returns any output. Not sure if this is required or optional. It however does not seem to be directly related to the bigger problem at hand, but it's worth noting.
I found the mistake!
When using writeTarball()
, the headers for each file are built manually:
Lines 136 to 153 in bd02047
Now, when we look at the tar
standard format, we see that the header byte 156 needs to contain char typeflag;
. This typeflag
is meant to be set here:
Line 146 in bd02047
Now in this case, the number (from ord()
) does not get converted to a char
; it returns nothing. That's why the typeFlag
is missing
Line 99 in bd02047
tarball.contents
Lines 109 to 114 in bd02047
Tarball()
is never actually extracted:Lines 201 to 220 in bd02047
Tried updating my nitter instance, which updated zippy since then. Getting instruction aborts in inflateBlock. Rolling back to 0.9.11 works fine.
localhost:~/zippy# nimble --version
nimble v compiled at 2022-08-05 03:01:31
git hash: 43804b9c6ecec455a6851242a45a19b31b9b0776
localhost:~/zippy# nim --version
Nim Compiler Version 1.6.8 [Linux: amd64]
Compiled at 2022-11-15
Copyright (c) 2006-2021 by Andreas Rumpf
active boot switches: -d:release
localhost:~/zippy# gcc --version
gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924
Steps to reproduce:
localhost:~/zippy# nimble test -d:Release
Verifying dependencies for [email protected]
Compiling /root/zippy/tests/test (from package zippy) using c backend
randtest1.gz compressed: 4119 gold: 4096
randtest2.gz compressed: 8215 gold: 8192
randtest3.gz compressed: 12311 gold: 12288
rfctest1.gz compressed: 9370 gold: 34127
SIGILL: Illegal operation.
Error: execution of an external program failed: '/root/zippy/tests/test '
Tip: 1 messages have been suppressed, use --verbose to show them.
Error: Execution failed with exit code 1
... Command: /usr/bin/nim c --noNimblePath -d:NimblePkgVersion=0.10.7 --hints:off -d:Release -r --path:. /root/zippy/tests/test
Works fine in debug, which suggest that there is some UB at play.
GDB points to this bit
literalsHuffman = initHuffman(fixedLitLenCodeLengths)
distancesHuffman = initHuffman(fixedDistanceCodeLengths)
else:
let
> hlit = b.readBits(5).int + 257
hdist = b.readBits(5).int + 1
hclen = b.readBits(4).int + 4
It seems that mummy is using this library to automatically perform on-the-fly gzip compression for HTTP responses which is great for performance, but in certain scenarios can be a serious security vulnerability due to BREACH
The BREACH link mentions HTB (Heal-the-BREACH) as an inexpensive, but effective mitigation. It would be nice if this is implemented, so that gzip compression can be used safely without people having to worry about the specifics of the application, threat model, etc.
thanks
By reading the source, I've discovered I can disable simd with -d:zippyNoSimd
so this isn't a blocker for me anymore, but in case it's fixable, I've filed this anyway. Feel free to close if it's not worth the time.
# samp.nim
import zippy
let gz = compress("something")
doAssert uncompress(gz) == "something"
Compiling samp.nim
with this command:
nim c --cpu:arm64 --threads:on --parallelBuild:1 --tlsEmulation:off "--passC:-arch arm64" "--passC:-isysroot" "--passC:/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.2.sdk" /tmp/samp.nim
Results in the error Cannot select: intrinsic %llvm.aarch64.crc32b
:
Hint: clang -c -w -ferror-limit=3 -pthread -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.2.sdk -I/Users/matt/.choosenim/toolchains/nim-1.6.6/lib -I/private/tmp -o /Users/matt/.cache/nim/samp_d/@m..@s..@sUsers@[email protected]@[email protected]@szippy@scrc32_simd.nim.c.o /Users/matt/.cache/nim/samp_d/@m..@s..@sUsers@[email protected]@[email protected]@szippy@scrc32_simd.nim.c [Exec]
fatal error: error in backend: Cannot select: intrinsic %llvm.aarch64.crc32b
clang: error: clang frontend command failed with exit code 70 (use -v to see invocation)
Apple clang version 11.0.0 (clang-1100.0.33.17)
Target: aarch64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
clang: note: diagnostic msg: PLEASE submit a bug report to http://developer.apple.com/bugreporter/ and include the crash backtrace, preprocessed source, and associated run script.
clang: note: diagnostic msg:
********************
PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /var/folders/mt/84ydn6s16y793cksy0ykq7zw0000gn/T/@m-2a6dff.c
clang: note: diagnostic msg: /var/folders/mt/84ydn6s16y793cksy0ykq7zw0000gn/T/@m-2a6dff.sh
clang: note: diagnostic msg: Crash backtrace is located in
clang: note: diagnostic msg: /Users/matt/Library/Logs/DiagnosticReports/clang_<YYYY-MM-DD-HHMMSS>_<hostname>.crash
clang: note: diagnostic msg: (choose the .crash file that corresponds to your crash)
clang: note: diagnostic msg:
********************
Error: execution of an external program failed: 'clang -c -w -ferror-limit=3 -pthread -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.2.sdk -I/Users/matt/.choosenim/toolchains/nim-1.6.6/lib -I/private/tmp -o /Users/matt/.cache/nim/samp_d/@m..@s..@sUsers@[email protected]@[email protected]@szippy@scrc32_simd.nim.c.o /Users/matt/.cache/nim/samp_d/@m..@s..@sUsers@[email protected]@[email protected]@szippy@scrc32_simd.nim.c'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.