michaeleisel / zippyjson

A much faster version of JSONDecoder

ZippyJSON

A much faster version of JSONDecoder

Benchmarks

These benchmarks were done on a Macbook Pro. The results are very similar on the iPhone (ZippyJSON is 3x+ faster for all 3 files on both platforms).

Usage

Just replace JSONDecoder with ZippyJSONDecoder wherever you want to use it. So instead of let decoder = JSONDecoder(), do let decoder = ZippyJSONDecoder(), and everything will just work. This is because ZippyJSONDecoder has the exact same API as JSONDecoder (i.e. it's drop-in). Also, don't forget to add import ZippyJSON in files where you use it.

NOTE: when measuring the speed of ZippyJSON, make sure you're building for release

Why is it so much faster?

• Apple's version first converts the JSON into an NSDictionary using NSJSONSerialization and then afterwards makes things Swifty. The creation of that intermediate dictionary is expensive.
• ZippyJSON is built largely in C++ (but still with a Swift interface wrapped around it). For the initial parsing (you might call it tokenizing), it uses simdjson, a very fast library that makes good use of vectorization. Apple, on the other hand, uses entirely Swift (aside from the use of NSJSONSerialization) which is generally slower.
• There are many specific optimizations in there as well. For example, date parsing for ISO-8601 dates is 10x faster due to using JJLISO8601DateFormatter instead of Apple's date formatter.

So, it's largely due to Apple trying to be elegant and operate at a higher level.

When should you use this library?

At first, default to using JSONDecoder. It's very battle-tested, and for plenty of use cases is just fine. Then, once you start looking for new things to optimize, take a look at how long your JSON parsing is taking. After all, JSON parsing can be a bottleneck for getting data to the user. As a rule of thumb, divide its current time taken by 4 to approximate the time taken with ZippyJSON. If that difference is significant to you (and even milliseconds can impact a user experience!), then consider using ZippyJSON.

Future improvements

There are still many places in the code that are ripe for optimization. Feel free to submit a ticket if you have a specific case where you need more performant JSON parsing, and where ZippyJSON is not already 4x faster than Apple's. JSONEncoder and NSJSONSerialization are also promising for optimzation, please chime in if you need one of these improved.

Installation

Cocoapods

ZippyJSON is available through CocoaPods (SPM support is in the works). To install it, simply add the following line to your Podfile:

pod 'ZippyJSON'

You can also make it pod 'ZippyJSON', :inhibit_warnings => true if you want to suppress all warnings.

SwiftPM (iOS only)

Add the package in the SwiftPM packages area with repository URL https://github.com/michaeleisel/ZippyJSON

Author

Michael Eisel, [email protected]

zippyjson's Issues

ISO8601 with fractional seconds?

Hi,

I need to parse remote date encoded as iso8601 with fractional seconds. However, using something like:

        let decoder = ZippyJSONDecoder()
        decoder.dateDecodingStrategy = .formatted(ISO8601withFractionalSeconds())

Seems to be as slow as classic JSON, so I guess it fallbacks to default when using formatted as well as custom ? How can I proceed?

Thanks!

Fragment Decoding

When trying to decode a Codable primitive value at root level, these errors are thrown:

Error thrown by ZippyJSONDecoder: "The data couldn’t be read because it isn’t in the correct format."
Error thrown by JNT: "The given data was not valid JSON. Error: Problem while parsing a string"

As of iOS 13.1+ and macOS 10.15.1+, JSONDecoder is capable of decoding Codable primitive values, so my tests for encoding/decoding primitive values pass with the standard JSONDecoder, but fail with ZippyJSONDecoder.

func testExample() {
    enum ExampleEnum: String, Codable {
        case value
    }

    let codablePrimitive: ExampleEnum = .value

    do {
        let data = try JSONEncoder().encode(codablePrimitive)
        let decoded = try JSONDecoder().decode(ExampleEnum.self, from: data)
        let zippyDecoded = try ZippyJSONDecoder().decode(ExampleEnum.self, from: data)

        XCTAssertEqual(decoded, codablePrimitive)
        XCTAssertEqual(zippyDecoded, codablePrimitive)
    }
    catch {
        XCTFail(error.localizedDescription)
    }
 }

Fix issue with building from SwiftPM

Support for .xcodeproj with Carthage, and not just pre-built frameworks

Example demo application.

Firstly, Thank you for your attempt in area of JSON parsing smooth & faster by adapting robust functionality of C & C++.

I am using Swift 5.0 as Primary programming language for my iOS application.
I am not sure, How would I make use of this library.
However, I have integrated this library into my demo project using Cocoapods.
But, don't exactly know How would I make use of this library.
It would be fruitful to have little demo app that showcase the utilisation of this library.

Thank you.
Rashesh

Carthage Errors

I get the following errors when trying to update Carthage with ZippyJSON in my Cartfile for a macOS project:

*** Skipped building ZippyJSONCFamily due to the error:
Dependency "ZippyJSONCFamily" has no shared framework schemes for any of the platforms: Mac

*** Skipped building ZippyJSON due to the error:
Dependency "ZippyJSON" has no shared framework schemes for any of the platforms: Mac

"Use of undeclared identifier 'vpaddq_u8'" error when building with Release config and using SPM

Hi,
This library fails to build with the Release configuration when it is installed in my project with the Swift Package Manager. When using Cocoapods this doesn't happen.

There are a few different errors, all in the simdjson.h file, about the use of undeclared identifiers.

SPM support

Any chance you could add SPM support? Now that SPM is build into Xcode it will probably grow into the most used package manager for Swift projects so it would be great to support it.

I am currently working on a side project where I need to parse multiple GB of json data and I would love to use this library but the project uses SPM as a dependency manager.

not able to parse an Int value that came as nil

do you only parse as string and Bool?

var price: Int?

Swift.DecodingError.dataCorrupted(Swift.DecodingError.Context(codingPath: [LazyJSONKey(stringValue: "Index 0", intValue: 0), LazyJSONKey(stringValue: "price", intValue: nil)], debugDescription: "Parsed JSON number 2500 does not fit.", underlyingError: nil))

Unknown type name '__uint128_t' on Xcode 12

Hi, i've got an archiving error on ZippyJSON when archiving my app project, in the simdjson.cpp file :

really_inline value128 full_multiplication(uint64_t value1, uint64_t value2) {
  value128 answer;
#ifdef _MSC_VER
  // todo: this might fail under visual studio for ARM
  answer.low = _umul128(value1, value2, &answer.high);
#else
  __uint128_t r = ((__uint128_t)value1) * value2;
  answer.low = r;
  answer.high = r >> 64;
#endif
  return answer;
}

I have no idea what changed between xcode versions but i had to rollback to using the UIKit regular JSONDecoder.

So many warnings in Xcode 11.2

I am seeing so many warnings in Xcode 11.2.

How can we fix that?

Update simdjson to 0.7

Simdjson 0.7 has now been released. Time to update the library?

Consider updating to 0.4.0

Version 0.4.0 of simdjson is now available

Highlights

• Test coverage has been greatly improved and we have resolved many static-analysis warnings on different systems.

New features:

• We added a fast (8GB/s) minifier that works directly on JSON strings.
• We added fast (10GB/s) UTF-8 validator that works directly on strings (any strings, including non-JSON).
• The array and object elements have a constant-time size() method.

Performance:

• Performance improvements to the API (type(), get<>()).
• The parse_many function (ndjson) has been entirely reworked. It now uses a single secondary thread instead of several new threads.
• We have introduced a faster UTF-8 validation algorithm (lookup3) for all kernels (ARM, x64 SSE, x64 AVX).

System support:

• C++11 support for older compilers and systems.
• FreeBSD support (and tests).
• We support the clang front-end compiler (clangcl) under Visual Studio.
• It is now possible to target ARM platforms under Visual Studio.
• The simdjson library will never abort or print to standard output/error.

[ZippyJSONDecoder] Warning: fell back to using Apple's JSONDecoder.

Hello @michaeleisel

Firstly I appreciate the time you've invested into OSS and wanna say thank you for this library.
Was playing a bit with 1.2.6 hoping to replace the default JSONDecoder and compare performance, but I'm getting this console warning:

[ZippyJSONDecoder] Warning: fell back to using Apple's JSONDecoder.
Reason: This library was not compiled with the necessary vector extensions (this is likely because you're using SwiftPM + the simulator, and is due to limitations with SwiftPM. This does not apply to real devices.). 
This message will only be printed the first time this happens. 
To suppress this message entirely, for all reasons, use `ZippyJSONDecoder.zjd_suppressWarnings = true

Wondering if it's a known & perhaps temporary issue, or some kind of limitation we have to accept and put up with.
Will it ever be possible to get benefits of this lib building/running on simulator?
Thanks in advance!

Array coded by Apple JSONCoder can't be decoded

Trying to decode an array of objects throws an error claiming the file isn't valid JSON. The file was created by the Apple JSON Coder, which can also decode it just fine.

The data in question:

[{"hi":1,"there":2,"comment":"comment"},{"hi":2,"there":4,"comment":"comment"},{"hi":3,"there":6,"comment":"comment"},{"hi":4,"there":8,"comment":"comment"},{"hi":5,"there":10,"comment":"comment"},{"hi":6,"there":12,"comment":"comment"},{"hi":7,"there":14,"comment":"comment"},{"hi":8,"there":16,"comment":"comment"},{"hi":9,"there":18,"comment":"comment"},{"hi":10,"there":20,"comment":"comment"}]

The error:

Fatal error: 'try!' expression unexpectedly raised an error: Swift.DecodingError.dataCorrupted(Swift.DecodingError.Context(codingPath: [], debugDescription: "The given data was not valid JSON. Error: Unexpected error, consider reporting this problem as you may have found a bug in simdjson", underlyingError: nil)): file /Users/marcel/programming/Examples/CodingTests/CodingTestsSwift/AppDelegate.swift, line 90

Code:

func readZippyCoder(data:Data) -> [TestClass] {
    NSLog("Zippy Decoding")
    let coder=ZippyJSONDecoder( )
    let array=try! coder.decode([TestClass].self, from: data)
    return array
}

Copy-pasted from the Apple Decoder code:

func readJSONCoder(data:Data) -> [TestClass] {
    NSLog("Swift Decoding")
    let coder=JSONDecoder( )
    let array=try! coder.decode([TestClass].self, from: data)
    return array
}

AppDelegate.swift

Issue with Cocoapods

It's strange, but I can't find ZippyJSON at Cocoapods.org anymore and the latest release (1.0.1) isn't available with this platform. Am I missing something?

Consider upgrading to simdjson 0.3.1

Version 0.3 of simdjson is now available

Highlights

• Multi-Document Parsing: Read a bundle of JSON documents (ndjson) 2-4x faster than doing it individually. API docs / Design Details
• Simplified API: The API has been completely revamped for ease of use, including a new JSON navigation API and fluent support for error code and exception styles of error handling with a single API. Docs
• Exact Float Parsing: Now simdjson parses floats flawlessly without any performance loss (simdjson/simdjson#558).
  Blog Post
• Even Faster: The fastest parser got faster! With a shiny new UTF-8 validator
  and meticulously refactored SIMD core, simdjson 0.3 is 15% faster than before, running at 2.5 GB/s (where 0.2 ran at 2.2 GB/s).

Minor Highlights

• Fallback implementation: simdjson now has a non-SIMD fallback implementation, and can run even on very old 64-bit machines.
• Automatic allocation: as part of API simplification, the parser no longer has to be preallocated-it will adjust automatically when it encounters larger files.
• Runtime selection API: We've exposed simdjson's runtime CPU detection and implementation selection as an API, so you can tell what implementation we detected and test with other implementations.
• Error handling your way: Whether you use exceptions or check error codes, simdjson lets you handle errors in your style. APIs that can fail return simdjson_result, letting you check the error code before using the result. But if you are more comfortable with exceptions, skip the error code and cast straight to T, and exceptions will be thrown automatically if an error happens. Use the same API either way!
• Error chaining: We also worked to keep non-exception error-handling short and sweet. Instead of having to check the error code after every single operation, now you can chain JSON navigation calls like looking up an object field or array element, or casting to a string, so that you only have to check the error code once at the very end.

Getting "One or more keys had non-ASCII characters"

I keep getting this error while my json doesn't appear to have no-ascii keys

https://gist.github.com/Kalzem/4c1483476693db06d4783d9243f68345

What can possibly trigger this error?

When I try to print the returned iterators on JNTCheck(iterator), this is what appears

Conformance to Combine.TopLevelDecoder

The proposal is to make ZippyJSONDecoder conform to TopLevelDecoder protocol from Combine framework. This will allow to use ZippyJSON in reactive publisher chaining:

URLSession.shared.dataTaskPublisher(for: request)
                    .map { $0.data }
                    .decode(type: Response.self, decoder: ZippyJSONDecoder()) // <----
                    .receive(subscriber: mySubscriber)

At first glance there is enough to add these lines to ZippyJSONDecoder.swift as Apple did with JSONDecoder:

@available(OSX 10.15, iOS 13.0, tvOS 13.0, watchOS 6.0, *)
extension ZippyJSONDecoder: TopLevelDecoder {
    public typealias Input = Data
}

Xcode Warning

Using the v1.2.1 with SwiftPM on Xcode v13.1, I got this console log msg:

[ZippyJSONDecoder] Warning: fell back to using Apple's JSONDecoder. Reason: This library was not compiled with the necessary vector extensions (this is likely because you're using SwiftPM + the simulator, and is due to limitations with SwiftPM. This does not apply to real devices.). This message will only be printed the first time this happens. To suppress this message entirely, for all reasons, use `ZippyJSONDecoder.zjd_suppressWarnings = true

Swift 5 Support

It seems that this project only supports Swift 5.1 and not Swift 5? I am unable to compile on Swift 5 due to missing returns as well as a No template named 'deque' in namespace 'std'.

Not enough bits to represent the passed value

On macOS, for a CLI utility, Running this on a large (30gb) JSON file and getting the following error:

Swift/Integers.swift:3444: Fatal error: Not enough bits to represent the passed value

Works fine with small test file of same struct (json sample and model below)

Main difference when using Big file vs test file is it gets loaded into virtual memory, JSONDecoder() doesn't seem to have an issue with the big file, however it uses absurd amounts of memory/disk space for the ginormous file, and is very slow, hence the desire to use ZippyJSON.

Also note: This is how the data comes to me, I can't just split it out... The purpose of this utility is just that ;) trying to split it out.

struct Report: Codable {
    let sensorAndUsageData: [SensorAndUsageDatum]
    enum CodingKeys: String, CodingKey {
        case sensorAndUsageData = "SensorAndUsageData"
    }
}
struct SensorAndUsageDatum: Codable {
  let accelerometer, gyroscope: [MotionSensorData]?
  var motionSensorData: [MotionSensorData] { accelerometer ?? (gyroscope ?? []) }
  enum CodingKeys: String, CodingKey {
    case accelerometer = "com.apple.SensorKit.motion.accelerometer"
    case gyroscope = "com.apple.SensorKit.motion.gyroscope"
  }
}
struct MotionSensorData: Codable {
    let metadata: Metadata
    let sample: [Sample]
}
struct Metadata: Codable {
    let device: Device
    let timestamp: Double
}
struct Device: Codable {
    let model: String
}
struct Sample: Codable {
  let acceleration: Motion?
  let start: Double
  let rotationRate: Motion?
  var motion: Motion { return acceleration ?? (rotationRate ?? Motion(x: 0, y: 0, z: 0))}
}
struct Motion: Codable {
  let x, y, z: Double
}

{"SensorAndUsageData": [{"accelerometer": [{"metadata":{
  "device" : {
    "model" : "iPhone"
  },
  "timestamp" : 27342999.47155517
},"sample":[{"acceleration": {"x": -0.857421875, "y": -0.50927734375, "z": 0.01171875}, "start": 654428838.25630069, "timestamp": 194207.30908599999, "identifier": 0},
{"acceleration": {"x": -0.8583984375, "y": -0.50927734375, "z": 0.01220703125}, "start": 654428838.26624668, "timestamp": 194207.319032, "identifier": 0},
{"acceleration": {"x": -0.85791015625, "y": -0.509521484375, "z": 0.011962890625}, "start": 654428838.27619267, "timestamp": 194207.32897800001, "identifier": 0},
{"acceleration": {"x": -0.8583984375, "y": -0.509033203125, "z": 0.01220703125}, "start": 654428838.2861377, "timestamp": 194207.338923, "identifier": 0}]}]},
{"gyroscope": [{"metadata":{
  "device" : {
    "model" : "iPhone"  
  },
  "timestamp" : 27342999.47155517
},"sample":[{"rotationRate": {"x": -0.857421875, "y": -0.50927734375, "z": 0.01171875}, "start": 654428838.25630069, "timestamp": 194207.30908599999, "identifier": 0},
{"rotationRate": {"x": -0.8583984375, "y": -0.50927734375, "z": 0.01220703125}, "start": 654428838.26624668, "timestamp": 194207.319032, "identifier": 0},
{"rotationRate": {"x": -0.85791015625, "y": -0.509521484375, "z": 0.011962890625}, "start": 654428838.27619267, "timestamp": 194207.32897800001, "identifier": 0},
{"rotationRate": {"x": -0.8583984375, "y": -0.509033203125, "z": 0.01220703125}, "start": 654428838.2861377, "timestamp": 194207.338923, "identifier": 0}]}]}]}

Xcode Swift packages support

Very interested in trying out your library in my iOS app. Trying to add ZippyJSON as a Swift package dependency, it gives me the following error in Xcode 11.

Showing All Messages
: the package zippyjson[https://github.com/michaeleisel/ZippyJSON.git] @ 1.0.0 contains incompatible dependencies:
zippyjsoncfamily[../ZippyJSONCFamily] @ local
jjliso8601dateformatter[../JJLISO8601DateFormatter] @ local

I also tried first adding the two other dependencies separately, but it still fails. Any ideas what could be causing this?

Update release notes with Release Optimisations

Hi 👋

This isn't necessarily an issue but a small suggestion.

I highly recommend that the README.md is updated to mention how much this framework benefits from Release optimisations.

In my personal case, I was looking into adding it to a personal app and after the initial results I was quite disappointed as ZippyJSONDecoder was actually taking almost 100% more time than JSONDecoder. It was only, after I stumbled in a few issues that I saw it was recommended to test it in a Release build, and, after building for Release instead of Debug, I actually saw the power of ZippyJSONDecoder by decreasing the time it took to boot my app from 1.5s to 0.84s.

Version 1.1: attempted free of pointer that was not claimed by any registered malloc zone

This crash started happening in version 1.1, but did not happen for 1.0 or 1.0.1. I decode an array into two different types depending on which one matches the value.

Crash at ***

ZippyJSONDecoder:

if isKnownUniquelyReferenced(&wrapper.decoder) {
    let innerDecoder = wrapper.decoder
    *** JNTReleaseValue(innerDecoder.value)
    (innerDecoder.value, innerDecoder.isEmpty) = try JSONKeyedDecoder<Key>.setupValue(value, decoder: decoder, convertToCamel: convertToCamel)
    return KeyedDecodingContainer(wrapper.decoder)
}

JSONSerialization.mm:

void JNTReleaseValue(DecoderPointer decoder) {
    *** delete decoder;
}

JSON:

"option": [
  "24\" x 84\" (2x7)",
  "36\" x 120\" (3x10)",
  "36\" x 60\" (3x5)",
  {
     "option": "Other (Please Specify)",
     "value": "bannerSizeOther"
  }
],

in my code: single value decoding to either FormDriver or String:
edited to remove proprietary code

in my code: FormDriver/String array decoding
edited to remove proprietary code

Faster float parsing

Right now, it uses strtod, which is fairly slow. We can do better, let's see what the result of simdjson/simdjson#242 is.

Having a string with a missing end-quote hard crashes instead of returning error

In https://github.com/michaeleisel/ZippyJSONCFamily/blob/c7fa94c7983fa9becc4f8af0e42560d9fc80571f/Sources/ZippyJSONCFamily/simdjson.cpp#L32 you're missing case 14: UNCLOSED_STRING, so when ZippyJSON tries to get the error msg it crashes.

The test case I was seeing it with is:

{
    "key: "yes"
}

Note the missing "

You also have a typo for UNESCAPED_CHARS, 'escapted' instead of 'escaped' twice.

First time of ZippyDecoding seems no improvement of CPU time cost

For a json file, if we only run once of decoding, it seems no improvement of CPU time cost when comparing with JSONDeocoder.

Background:

My app have more than 100 json files, and each file is a unique Model. Total time cost with JSONDeocder is about 2.0 seconds, and with ZippyJSONDecoder is about 2.5 seconds, so Zippy performs even worse.

Analysis:

I did some tests with ZippyJSON's test cases, and find same result.

Example:
Modify file Tests/Models/github_events.swift, replace all Date virables to String, such as changer let createdAt: Date to let createdAt: String , in order to ignore the affect of Date decoding, as README said:

date parsing for ISO-8601 dates is 10x faster due to using JJLISO8601DateFormatter instead of Apple's date formatter.

then run run("github_events", ghEvents.self, dateDecodingStrategy: .iso8601)

And print all 10 times data , no only focus on the final average time.

// ZippyJSONDecoder
[0.0027, 0.0012, 0.0012, 0.0012, 0.0012, 0.0011, 0.0008, 0.0008, 0.0008, 0.0008]

// JSONDecoder
[0.0031, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0022, 0.0020, 0.0020, 0.0020]

So, if we only compare the first time, 0.0027 vs 0.0031, seems no 3x faster.

And I find below logic in ZippyJSONDecoderTests.swift

func averageTime(_ closure: () -> ()) -> CFTimeInterval {
let count = 10
var times: [CFTimeInterval] = []
for _ in 0..<count {
times.append(time(closure))
}
return times.dropFirst(count / 3).reduce(0, +) / CFTimeInterval(times.count)
}

As code shows, when calculating average time, we dropped first 3 cases, exactly cases that perform not so good.

Questions:

Why we need to drop first 3 cases?
And why the first time has no 3x improvement? any way to optimize this?

Best wishes~

Dictionary representation of JSON

My app represents JSON in tree view(NSOutlineView). So, apparently my app does not have any concrete types hence I cannot use JSONDecoder
Can I use ZippyJSON for functionality which would be similar to NSJSONSerialization which would get me a native Swift dictionary representation?

Incorrect 2d array parsing ?

I replace JSONDecoder with ZippyJSONDecoder in Lottie library but it stops decode correctly. I get error:

typeMismatch(Swift.Array<Any>, Swift.DecodingError.Context(codingPath: [JSONKey(stringValue: "v", intValue: nil)], debugDescription: "Tried to unbox array, but it wasn\'t an array", underlyingError: nil))

I tried to parse this file: Lottie file
There are 2 cases where "v" is a key:

• "v":"5.5.6"
• "v":[[-156.5,-406],[-166.5,-367],[-146.5,-327],[-166.5,-286],[-146.5,-246],[-166.5,-206],[-146.5,-165],[-166.5,-127],[-156.5,-84]]

My fork of Lottie library with ZippyJSON

Using ZippyJSON with Swift Packages Fails (outdated `Package.swift` file)

The Failure seems to be down to just the Package.swift file getting an old verson from tag 1.2.1 the File looks like this in the tag:

// swift-tools-version:5.1
import PackageDescription
let package = Package(
    name: "ZippyJSON",
    platforms: [
        .iOS(.v11),
    ],
    // snip

This will fail building with:

JSONDecoder.swift:20:32: ‘withInternetDateTime’ is only available in macOS 10.12 or newer

Using the master branch in with Swift package gets the correct Package.swift file and builds successfully 🎉

Please tag and release the current files in the repository as a version so we do not have to use master in Swift Packages.

Thanks!

Reduce size impact of ZippyJSON

Although I haven't had any reports of ZippyJSON causing a significant impact on app size, there's probably some low-hanging fruit lying around size that I could improve. There are two sorts of size contributions: that of the library itself, and (maybe?) from each individual invocation and inlining that could occur for it.

ZippyJSON is slower than Apple JSONDecoder sometimes

Device

iPhone XS

iOS

13.3.1

Swift object

struct User: Decodable {
    let firstName: String
    let lastName: String
    let email: String
    let login: String
    let passwordHash: String
    let gender: String
    let avatarUrl: String
    let country: String
    let city: String
    let zipCode: Int
    let phone: String
    let isVip: Bool
    let isFamous: Bool
}

JSON

1. 10 users
2. 100 users
3. 1K users

Results

JSONDecoder is 20-35% faster than ZippyJSONDecoder

Data

Link to gist

Compatibility with iOS 10.x

Hi!

Currently, ZippyJSON cannot be added to iOS projects where the deployment target is less than 11.0 – the version that's specified in the Podspecs of ZippyJSON, ZippyJSONCFamily and JJLISO8601DateFormatter alike. (Even though the latter's README says it should be iOS 10+.)

But at the same time, there are availability checks that look for iOS 10.0, mostly around the ISO8601 date strategy, that are handled gracefully... (Which isn't necessary if the target is 11.0 anyway.)

Do you think it's possible to lower the deployment target to 10 or below?
Are you aware of any issues that would block such a move?

I'd really like to try using it in a more legacy-ish project :)

Thanks for this very promising library, by the way.

Xcode warnings

I'm getting these Xcode warnings in my log after building.

ld: warning: direct access in function 'simdjson::haswell::implementation::stage2(unsigned char const*, unsigned long, simdjson::dom::parser&) const' from file '/Build/Products/Debug-iphonesimulator/ZippyJSONCFamily.o' to global weak symbol 'typeinfo name for std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> >' from file '/Build/Products/Debug-iphonesimulator/ZippyJSONCFamily.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
ld: warning: direct access in function 'simdjson::haswell::implementation::stage2(unsigned char const*, unsigned long, simdjson::dom::parser&) const' from file '/Build/Products/Debug-iphonesimulator/ZippyJSONCFamily.o' to global weak symbol 'typeinfo name for std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> >' from file '/Build/Products/Debug-iphonesimulator/ZippyJSONCFamily.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
ld: warning: direct access in function 'simdjson::haswell::implementation::stage2(unsigned char const*, unsigned long, simdjson::dom::parser&, unsigned long&) const' from file '/Build/Products/Debug-iphonesimulator/ZippyJSONCFamily.o' to global weak symbol 'typeinfo name for std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> >' from file '/Build/Products/Debug-iphonesimulator/ZippyJSONCFamily.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
ld: warning: direct access in function 'simdjson::haswell::implementation::stage2(unsigned char const*, unsigned long, simdjson::dom::parser&, unsigned long&) const' from file '/Build/Products/Debug-iphonesimulator/ZippyJSONCFamily.o' to global weak symbol 'typeinfo name for std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> >' from file '/Build/Products/Debug-iphonesimulator/ZippyJSONCFamily.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.

Fix SwiftUI Previews (support arm64 on simulators)

This library is breaking SwiftUI Previews.

Error:
/x.swift:9:8: Could not find module 'ZippyJSON' for target 'arm64-apple-ios-simulator'; found: x86_64-apple-ios-simulator, at: /Users/x/Library/Developer/Xcode/DerivedData/CricHQ-dmjqadgxdcinrrcgwugqdedztquy/Build/Products/Debug-iphonesimulator/ZippyJSON.swiftmodule

ZippyJSON 1.2.5 depends on JJLISO8601DateFormatter 0.1.4 which does not exist

Hi, thanks for a great work!

Faced a problem while using ZippyJSON '~> 1.2.5' which requires JJLISO8601DateFormatter version 0.1.4. But current JJLISO8601DateFormatter podspec has 0.1.3 version whereas the last JJLISO8601DateFormatter release is only 0.1.2

Empty strings are seen as +Infinity, instead of throwing even when nonConformingFloatDecodingStrategy == .throw

This is because here:

ZippyJSON/Sources/ZippyJSON/ZippyJSONDecoder.swift

Line 66 in c380ac9

return JNTCreateContext(string, UInt32(length), "", "", "")

JNTCreateContext is called with empty strings when nonConformingFloatDecodingStrategy == .throw

"" is then interpreted here:

ZippyJSON/Carthage/Checkouts/ZippyJSONCFamily/Sources/ZippyJSONCFamily/JSONSerialization.mm

Lines 365 to 384 in c380ac9

	double JNTDocumentDecode__Double(DecoderPointer decoder) {
	if (TypeChecker::Double(&decoder->iterator)) {
	return Converter::Double(&decoder->iterator);
	} else if (decoder->iterator.is_integer()){
	return (double)decoder->iterator.get_integer();
	} else {
	if (decoder->iterator.is_string()) {
	const char *string = decoder->iterator.get_string();
	if (strcmp(string, decoder->context->posInfString.c_str()) == 0) {
	return INFINITY;
	} else if (strcmp(string, decoder->context->negInfString.c_str()) == 0) {
	return -INFINITY;
	} else if (strcmp(string, decoder->context->nanString.c_str()) == 0) {
	return NAN;
	}
	}
	JNTHandleWrongType(decoder, decoder->iterator.get_type(), "double/float");
	return 0;
	}
	}

as +Infinity.

My guess is that there's the stringForFloats variable that is intended to be passed on to simdjson:

ZippyJSON/Sources/ZippyJSON/ZippyJSONDecoder.swift

Lines 344 to 349 in c380ac9

	switch (nonConformingFloatDecodingStrategy) {
	case .convertFromString(positiveInfinity: _, negativeInfinity: _, nan: _):
	stringsForFloats = true
	case .throw:
	stringsForFloats = false
	}

but that never happened.

Remove any ambient slowdowns in the Swift code

The C++ is pretty fast and the performance characteristics are well understood. The Swift layer, however, is a big question mark. For example, moving from struct to final class was a big help. I have no further ideas, so this is an open call for anyone to look at ZippyJSONDecoder.swift and see what they can find. For example, I think we could do less retains and releases.

Not faster than JSONDecoder for specific case

This issue is to track a case that was reported where ZippyJSON is only 7% faster than JSONDecoder

ZippyJSON doesn't throw errors until after a value has been unboxed, which means you can't use "try?" effectively

Hey there! I was integrating ZippyJSON into one of my projects, and I noticed that errors aren't thrown at the right time. I made a sample project demonstrating the issue here: https://github.com/noahsark769/ZippyJsonProject

The issue is that when the __JSONDecoder unboxes a value, it doesn't throw and always returns a default value, but then inspects JNTError later to throw. This means that code like this doesn't work:

let string = "{\"key\": 123}"

struct Example: Codable {
    let key: String

    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        self.key = (try? container.decode(String.self, forKey: .key)) ?? ""
    }
}

let decoder = ZippyJSONDecoder()
do {
    let data = string.data(using: .utf8)!
    let element = try decoder.decode(Example.self, from: data)
    print(element)
} catch {
    print(error)
}

This will catch the error and print:

typeMismatch(Any, Swift.DecodingError.Context(codingPath: [JSONKey(stringValue: "key", intValue: nil)], debugDescription: "Expected PKc value but found Number instead.", underlyingError: nil))

This is an issue for me since I'm parsing JSON where one of the keys can be either a string or a number, and I'm not sure that there's a way to implement this with ZippyJSON right now. (Note: Swift's JSONDecoder handles this case correctly.)

I think the fix would be to throw an error directly from __JSONDeocder: even though we'd have to check JNTError in every call to unbox, this would technically be more correct and solve the above issue (I think).

Let me know if you have any further thoughts @michaeleisel - I'm happy to submit a pull request at some point soon!

Cannot compile with Xcode 14.1 and 14.2 due to missing JNTErrorInfo and JNTGetErrorInfo

I'm seeing the following build failure when attempting to build ZippyJSON version 1.2.9 using Xcode 14.1 and Xcode 14.2 RC1:

/Users/tonyarnold/Developer/Repositories/ZippyJSON/Sources/ZippyJSON/ZippyJSONDecoder.swift:228:20: error: cannot find type 'JNTErrorInfo' in scope
    var errorInfo: JNTErrorInfo = JNTErrorInfo()
                   ^~~~~~~~~~~~
/Users/tonyarnold/Developer/Repositories/ZippyJSON/Sources/ZippyJSON/ZippyJSONDecoder.swift:229:5: error: cannot find 'JNTGetErrorInfo' in scope
    JNTGetErrorInfo(context, &errorInfo)
    ^~~~~~~~~~~~~~~

Linux compatibility

Right now, it only works with iOS and MacOS. Some things will need to be reworked for Linux (it basically can't use Objective-C)

Failed to decode Decimal value

The following snippet reproduces the issue.

struct Model: Codable {
    let amount: Decimal
}

let json = """
{
  "amount": 12.34
}
"""

let jsonDecoder = ZippyJSONDecoder()
do {
    let model = try jsonDecoder.decode(Model.self, from: json.data(using: .utf8)!)
}
catch {
    print(error)
}

Results:

dataCorrupted(Swift.DecodingError.Context(codingPath: [LazyJSONKey(stringValue: "amount", intValue: nil)], debugDescription: "Invalid Decimal", underlyingError: nil))

iOS Version: 15.4.1
iOS Device: iPhone 13 Pro Max

Are there dependencies blocking this library working on Linux?

Would like to use this on the server, and was wondering if there are any blockers. Thanks.

Warning: load of misaligned address

Hi,

on MacOS (12.2.1, m1), i have this warning when using ZippyJSON:

SourcePackages/checkouts/ZippyJSONCFamily/Sources/ZippyJSONCFamily/JSONSerialization.mm: runtime: Undefined Behavior: Load of misaligned address 0x0001100566dd for type 'const uint32_t' (aka 'const unsigned int'), which requires 4 byte alignment

after some digging, i found that it seems to be an issue in simdjson and was fixed by this pr:
simdjson/simdjson#1037

`michaeleisel/ZippyJSONCFamily` is missing commit for v1.2.4

x-posting this issue here (sorry, I didn't realize that michaeleisel/ZippyJSONCFamily wasn't actively watched!)

See also: michaeleisel/ZippyJSONCFamily@c55123b

---

From xcodebuild when pointing at SPM package michaeleisel/[email protected]:

Not sure if a commit didn't get pushed, while the tag did!

Custom key decoding strategy performance

Right now, custom key decoding strategies cause ZippyJSON to fall back to apple's decoder, meaning there's no perf win. This issue is to gauge how often people need custom key decoding, why they need it, etc. to see if it should be supported, and if so, how.

So please let me know: do you use custom key decoding? What do you need it for, and why?

Cannot get next value -- unkeyed container is at end

I am trying to extend Lottie-Ios with Zippy Json.
But when i use standard init: try ZippyJSONDecoder().decode(Animation.self, from: json)
It throws exception:

DecodingError
▿ valueNotFound : 2 elements
- .0 : Any
▿ .1 : Context
▿ codingPath : 2 elements
▿ 0 : JSONKey(stringValue: "layers", intValue: nil)
- stringValue : "layers"
- intValue : nil
▿ 1 : JSONKey(stringValue: "Index 1", intValue: 1)
- stringValue : "Index 1"
▿ intValue : Optional
- some : 1
- debugDescription : "Cannot get next value -- unkeyed container is at end."
- underlyingError : nil

JSON Attached
SpinnerLoader.json.zip

Feature request: Add more flexible usage abilities, not for only Decodable (NSJSONSerialization)

I know that pretty much folks not using Decodable protocol because in case of model properties names in Swift and JSON responses are different, you'll have to write large CodingKey enums.

To avoid this, to add more flexibility to framework users, I think it would be cool to add JSON deserialization implementation with NSJSONSerialization class.

What do you think?

Thanks you in advance