mw99 / datacompression Goto Github PK

Always compressed level 5 is too rigid

Your DataCompression.swift is exactly the kind of utility that I need to unzip downloaded files in a mobile App. The files are sqlite database files that are about 6MB compressed and 22MB uncompressed. I have tested that each of your compression algorithms will compress and decompress this data correctly. However, when I compress a file with MacOS command-line zip.

$ zip myFile.zip myFile

None of your decompression algorithms is able to decompress the 'myFile.zip'.

.inflate() and .ZLIB return the error 'Invalid BTYPE 3 block'. All of the others return no error. They all output a nil output Data.

Are there any settings that can be tweaked to solve that problem? Thanks so much for any help you can provide. I would really like to be able to use your code.

Below is the sample code that I have used to test this:

private func zip_unzip(sourceFile: String, targetFile: String, doZip: Bool) -> Void {
do {
let sourceUrl = URL(fileURLWithPath: NSHomeDirectory() + sourceFile)
print("source URL (sourceUrl)")
let source = try Data(contentsOf: sourceUrl)
print("source Data (source)")
let target: Data? = (doZip) ? source.compress(withAlgorithm: .ZLIB) : source.decompress(withAlgorithm: .ZLIB)
print("target Data (String(describing: target))")
let targetUrl = URL(fileURLWithPath: NSHomeDirectory() + targetFile)
print("target url (targetUrl)")
try target?.write(to: targetUrl, options: Data.WritingOptions.atomic)
} catch {
print("caught exception did not complete zip/unzip")
}
}

When using carthage as a dependency manager, i receive following Error if I try to upload my build

ERROR ITMS-90056: "This bundle Payload/..../Frameworks/DataCompression.framework is invalid. The Info.plist file is missing the required key: CFBundleVersion. Please find more information about CFBundleVersion at https://developer.apple.com/documentation/bundleresources/information_property_list/cfbundleversion"
Return status of iTunes Transporter was 1: ERROR ITMS-90056: "This bundle Payload/..../Frameworks/DataCompression.framework is invalid. The Info.plist file is missing the required key: CFBundleVersion. Please find more information about CFBundleVersion at https://developer.apple.com/documentation/bundleresources/information_property_list/cfbundleversion"
The call to the iTMSTransporter completed with a non-zero exit status: 1. This indicates a failure.

Apple convert single files to .zip when running "Compress this file", and would be good to be able to unzip the file. .unzip() isn't for this case. For example, Perfetto only has gzip support. I have C++ code to handle this, but was just missing this omission in this otherwise amazing library.

I ended up renaming .zip()/.unzip() -> .zlib()/unzlib() in my local copy.

I have been able to get compression and decompression to work just fine with every algorithm except the ones pertaining to ZLIB. So pretty much just deflate/inflate and compress/decompress with .ZLIB. I am taking a URL, getting it's data representation and passing it into the compression method and then posting it to the server. For decompression I am pulling the data down from the server, passing it into the decompression method and the creating a URL with the decompressed data. Like I said this process has worked every time as long as I am not using ZLIB. I am pretty much trying to see if anyone has had this problem in the past. Thanks.

I just added DataCompression to a Swift 5 based SwiftPM tool this one but I am now getting errors stating:

'compression_stream_init' is only available on OS X 10.11 or newer
'compression_stream_destroy' is only available on OS X 10.11 or newer

Here's my Package.swift file content:

// swift-tools-version:5.0

import PackageDescription

let package = Package(
    name: "Accio",
    platforms: [.macOS(.v10_12)],
    products: [
        .executable(name: "accio", targets: ["Accio"]),
        .library(name: "AccioKit", type: .dynamic, targets: ["AccioKit"])
    ],
    dependencies: [
        .package(url: "https://github.com/mw99/DataCompression.git", .upToNextMajor(from: "3.4.0")),
        .package(url: "https://github.com/Flinesoft/HandySwift.git", .upToNextMajor(from: "2.8.0")),
        .package(url: "https://github.com/onevcat/Rainbow.git", .upToNextMajor(from: "3.1.4")),
        .package(url: "https://github.com/jakeheis/SwiftCLI.git", .upToNextMajor(from: "5.2.2")),
        .package(url: "https://github.com/kareman/SwiftShell.git", .upToNextMajor(from: "4.1.2")),
        .package(url: "https://github.com/mengxiangyue/xcodeproj.git", .branch("master")), // TODO: switch back to original once https://github.com/tuist/xcodeproj/issues/385 is fixed
    ],
    targets: [
        .target(
            name: "Accio",
            dependencies: ["AccioKit"]
        ),
        .target(
            name: "AccioKit",
            dependencies: [
                "DataCompression",
                "HandySwift",
                "Rainbow",
                "SwiftCLI",
                "SwiftShell",
                "xcodeproj",
            ]
        ),
        .testTarget(
            name: "AccioKitTests",
            dependencies: ["AccioKit", "HandySwift", "xcodeproj"]
        )
    ]
)

Hey, first of all thanks for this great library! I use this in an app of mine- it seems to work very well.

However, I suddenly seem to get test failures (macOS 10.4.4, Xcode 10.2).

The falling test is testRandomDataBlob, specifically the .lzfse decompression test. Apples data compression lib seems to make it hard to find out what the actual error is though. Do you have any idea whats up with that?

Btw the way this test is set up makes it look like more tests are failing after a first test failure. However really only lzfse decompression fails.

I have problems using gunzip with Swift 5, Xcode 11.3.1

It doesn’t return the inflated content, instead I ‘ve got back the same deflated input data. Executed under test environment or common debug conditions in Xcode. Therefore in the playground environment the same code works perfect as expected!

The file gzipped XML file is about 36kb, inflated expected to be about 4MB

Are there some compiler settings I have to consider? Memory alignment? Any ideas?

I would like to know about the possibility of creating password protected zip files? what needs be done in terms of implementing this as an additional feature?

Thank you

Marcus,

I hate getting issues where someone says "there is a problem in your code..."
without giving details... so I hope you'll forgive my verbosity.

In C++, the following will generate a checksum of 0xA490238:

std::string test_string = "TEST CRC";
unsigned int a32 =  adler32(0L, (Bytef *)test_string.data(), test_string.length());

With the DataCompression module the same string generates a checksum of 0xA490239:

let test_crc = "TEST CRC"
var a32 = Adler32()
a32.advance(withChunk: test_crc.data(using:.utf8)!)
print ( "a32 = \(a32)" )

The issue appears to be related to the initialization of checksum for adler32:

--- DataCompression.swift:384 ---

/// Raw checksum. Updated after a every call to `advance(withChunk:)`
public var checksum:UInt32 = 1

--- DataCompression.swift:384 ---

In zlib/adler32, a 1L is returned only if the chunk being evaluated is null.
Otherwise, checksum starts with 0.

--- zlib-1.2.11/adler32.c:87 ---

 if (buf == Z_NULL)
        return 1L;

--- zlib-1.2.11/adler32.c:87 ---

Suggested fix:
initialize checksum to 0 (DataCompression.swift:385) and then,when returning checksum, return 1 if checksum == 0

--- DataCompression.swift:402 ---

    /// Formatted checksum.
    public var description: String
    {
        if checksum == 0 { // THB
            return "1"     // THB
        }                  // THB
        return String(format: "%08x", checksum)
    }

--- DataCompression.swift:402 ---

You would still have to account for checksum being read in other ways...

Thanks!

Tom Bowden
email: [email protected]

Been using this happily for years. Updated phone to iOS 16 and XCode to 14 today and now I'm getting fatalErrors in

func unzip(skipCheckSumValidation: Bool = true) -> Data?

Somewhere inside the

return last.withMemoryRebound(to: UInt32.self, capacity: 1)

statement. I don't get a lot of useful info from it, just

Swift/arm64-apple-ios.swiftinterface:23320: Fatal error
2022-09-14 16:03:58.690178-0700 TWiG MC[668:25066] Swift/arm64-apple-ios.swiftinterface:23320: Fatal error

in my console. My Data argument headed in was:

78.DA.0D.CD.3B.4E.43.31.10.40.51.B3.81.B4.94.D0.23.9E.C6.E3.4F.6C.29.EB.A0.45.E3.F9.C0.53.88.8D.EC.14.B0.FB.BC.F6.16.E7.3E.7D.5E.DC.29.55.2D.6C.31.A8.17.0B.A1.24.3A.37.23.00.89.39.86.AC.18.6B.33.03.C5.1C.19.53.40.CB.D2.72.80.7A.14.9F.C8.17.41.36.16.43.4E.44.29.85.5C.9B.A8.86.B3.C5.C4.B1.98.04.40.AA.15.98.35.A2.57.F6.26.02.10.92.02.D6.82.8D.51.01.DC.2F.AD.75.D5.7F.EF.5E.9A.B7.63.17.A9.94.06.87.AB.DF.63.DD.77.71.CF.DB.EB.C7.BB.07.78.EB.74.53.77.9A.8B.3A.FD.E9.DA.BA.FE.AC.D1.F7.39.F7.2F.BA.EF.A3.6F.3C.6E.AE.CD.71.D5.E9.DC.03.5C.71.40.90

I'll do some digging/experimenting. This has been rocksolid for so long, surprised to see it start misbehaving all of a sudden

Is there any plan to fix the deprecations for

'withUnsafeBytes is deprecated: use withUnsafeBytes<R>(_: (UnsafeRawBufferPointer) throws -> R) rethrows -> R instead

or is there a reasoning behind?

Thanks for the great work

any plans to update lib to Swift 4.2

I am using gzip with Compression of array of json [{ }, {}, {}...]
it compressed at iOS side and also decompress successfully at iOS side but when I upload compressed data to sever it fail to decompress.
even it fails here.
https://www.multiutil.com/text-to-gzip-compress/

I compress data and save it to document directory and then read compressed text then try to decompress text(array of json) by using https://www.multiutil.com/text-to-gzip-compress/ but it fails.

what could be issue?

Can you show a code example of unzipping a .gz folder and storing the folder on an ios device in order to access the files stored in the archive?

That would be highly appreciated.

Awful lot of warnings about deprecated uses of withUnsafeBytes..

Hi, I want to compress video file before uploading to server, should i use this? Does it work in my case?
Thanks

Thank you for providing us with a great library.

I am trying to create an application that uses this library.
I wanted to display the license clause of this library in my application.
However, I found the following in the LICENSE file

Perhaps this is an item that should be modified from the original template.

I would appreciate it if you could fix it if you could.

I got zlib compressed data from network and invoke data.inflate() to decompress.
It works fine in iOS 11 simulator, but fails in iOS 9 simulator.
The function compression_stream_process in DataCompression.swift always return -1.
Neither zlib level-5 nor level-10 can be decompressed
Same data can be decompressed by SWCompression in iOS9, so I think it's a bug in this library.

XCode Version 9.1 (9B55)
iOS 9.x simulator

I'm bring back some binary data from a web server using URLSession and POST.

let (data, response) = try await m_oURLSession.data(for: request)

after checking the content-encoding header is "deflate"

I run this:

inflatedData = data.inflate()

Which fails down deep and returns nil because stream.dst_size is not zero:

switch compression_stream_process(&stream, flags)
{
case COMPRESSION_STATUS_OK:
guard stream.dst_size == 0 else

I know the data is actually deflate because I've got an Android app which processes the same deflated data successfully.
Any ideas on what might be happening here?
Am I missing some conversion of data before calling inflate ?
TY

I would like to zip and unzip my files with a password if possible, I am not sure ! Thanks.

				if let compressedData = html.data.gzip() {
					try compressedData.write(to: filePath)
				}

I stored the compressData as into a SQLite database.
When I read it, it's a SQLite.Blob.
I tried but could not find a way to convert the back to .
Any idea?

I have been using this library until now but it will no longer run under Swift 5.7 because .withMemoryRebound has been updated.
https://github.com/apple/swift-evolution/blob/main/proposals/0333-with-memory-rebound.md

It would be great to get an update to be Swift 5.7 compatible.

Thanks

Build for Release failed because of Mantis:

ERROR ITMS-90056: "This bundle Payload/.../Frameworks/Mantis.framework is invalid. The Info.plist file is missing the required key: CFBundleVersion. Please find more information about CFBundleVersion at https://developer.apple.com/documentation/bundleresources/information_property_list/cfbundleversion"
Return status of iTunes Transporter was 1: ERROR ITMS-90056: "This bundle Payload/.../Frameworks/Mantis.framework is invalid. The Info.plist file is missing the required key: CFBundleVersion. Please find more information about CFBundleVersion at https://developer.apple.com/documentation/bundleresources/information_property_list/cfbundleversion"
The call to the iTMSTransporter completed with a non-zero exit status: 1. This indicates a failure.

I can't find the reason why CFBundleVersion is missing but by adding it manually inside /iOS/Carthage/Build/iOS/Mantis.framework fixed the problem temporary.

The following line in the license file should be filled by the copyright owner:

Copyright {yyyy} {name of copyright owner

I try to unzip my .zip file (downloaded from the Internet) using the .unzip () command and get a nil. Also I cannot open on my computer the file zipped by .zip(). With gzip all ok. Please help!

Is there a way to check if a given set of data is compressed in any way?
like data.isGzipped -> true or data.compression -> Compression.gzip || Compression.zip ,....

If not this would be a feature request

After compressing with:

let json = try? JSONSerialization.data(withJSONObject: sendData)
deflated = json?.compress(withAlgorithm: Data.CompressionAlgorithm.LZ4)

Error:
_lz4framed.Lz4FramedError, ('ERROR_frameType_unknown', 13)

inflated = lz4framed.decompress(data)

Perhaps different lz4 versions?

Python 3.6
lzframed: https://github.com/Iotic-Labs/py-lz4framed