quinnj / json3.jl

JSON3.jl

Documentation

Yet another JSON package for Julia; this one is for speed and slick struct mapping

TL;DR

Basic

# builtin reading/writing
JSON3.read(json_string)
JSON3.write(x)

# custom types
JSON3.read(json_string, T; kw...)
JSON3.write(x)

More complicated

# custom types: incrementally update a mutable struct
x = T()
JSON3.read!(json_string, x; kw...)
JSON3.write(x)

# read from file
json_string = read("my.json", String)
JSON3.read(json_string)
JSON3.read(json_string, T; kw...)

# write to file
open("my.json", "w") do f
    JSON3.write(f, x)
    println(f)
end

# write a pretty file
open("my.json", "w") do f
    JSON3.pretty(f, JSON3.write(x))
    println(f)
end

# generate a type from json
using StructTypes
JSON3.@generatetypes json_string_sample
JSON3.read(json_string, JSONTypes.Root)

json3.jl's Issues

Trouble reading in a file

This file https://raw.githubusercontent.com/jeffreylancaster/game-of-thrones/master/episode-character/episode-character.json does not load correctly with an error:

ERROR: ArgumentError: invalid JSON at byte position 1 while parsing type Any: InvalidChar

while JSON.jl reads it in without a problem.

updating read for StructTypes.Mutable()

I have a situation like that:

    Base.@kwdef mutable struct PublisherConfig
        CIName::String = "Test"
        CIFileName::String = "CI.yml"
        SSHSecretName::String = "SSH_private_access"
    end
    JSON3.StructType(::Type{PublisherConfig}) = JSON3.StructTypes.Mutable()
    const CONFIG = PublisherConfig()

Now, the assignment in the last line is important because it shall be a const. the const is important for performance. Thus, what I'm missing is an option to update the pre-existing object from JSON3 (rather than having JSON3 create a new one)

Inconsistent behaviour for NaNs and Infs

JSON standard only supports scientific decimal representation of floating point numbers. This leaves out Infs and NaNs. JSON3 currently represents these as string null when writing, and throws an error when trying to read them (into a struct). This behaviour needs to be consistent: either throw an error when trying to write something that is not supported by the standard, or augment the format by allowing NaN and Inf as floating point numbers. My preference would be the latter.

What do you think?

Reading and writing is broken for `Dict`s with struct keys

See the MWE below. I would have expected that when I read the object back in, the keys would have been Foo("a") and Foo("c"). Instead, they are Foo("Foo(\"a\")") and Foo("Foo(\"c\")"), which is not correct.

MWE:

julia> import JSON3

julia> import StructTypes

julia> mutable struct Foo
           value::String
           Foo() = new()
           Foo(value) = new(value)
       end

julia> mutable struct Bar
           value::String
           Bar() = new()
           Bar(value) = new(value)
       end

julia> mutable struct Baz
           dict::Dict{Foo, Bar}
           Baz() = new()
           Baz(dict) = new(dict)
       end

julia> StructTypes.StructType(::Type{Foo}) = StructTypes.Mutable()

julia> StructTypes.StructType(::Type{Bar}) = StructTypes.Mutable()

julia> StructTypes.StructType(::Type{Baz}) = StructTypes.Mutable()

julia> a = Baz(Dict{Foo, Bar}(Foo("a") => Bar("b"), Foo("c") => Bar("d")))
Baz(Dict{Foo,Bar}(Foo("a") => Bar("b"),Foo("c") => Bar("d")))

julia> keys(a.dict)
KeySet for a Dict{Foo,Bar} with 2 entries. Keys:
  Foo("a")
  Foo("c")

julia> values(a.dict)
ValueIterator for a Dict{Foo,Bar} with 2 entries. Values:
  Bar("b")
  Bar("d")

julia> b = JSON3.write(a)
"{\"dict\":{\"Foo(\\\"a\\\")\":{\"value\":\"b\"},\"Foo(\\\"c\\\")\":{\"value\":\"d\"}}}"

julia> c = JSON3.read(b, Baz)
Baz(Dict{Foo,Bar}(Foo("Foo(\"c\")") => Bar("d"),Foo("Foo(\"a\")") => Bar("b")))

julia> keys(c.dict)
KeySet for a Dict{Foo,Bar} with 2 entries. Keys:
  Foo("Foo(\"c\")")
  Foo("Foo(\"a\")")

julia> values(c.dict)
ValueIterator for a Dict{Foo,Bar} with 2 entries. Values:
  Bar("d")
  Bar("b")

cc: @quinnj

Bug in read Mutable not respecting overloaded JSON3.names

being careful about coercions of numeric types

Some of the type coercions that happen when reading numeric types are a bit too aggressive

julia> s = JSON3.write(typemax(UInt64))
"18446744073709551615"

julia> JSON3.read(s)
1.8446744073709552e19

julia> reinterpret(UInt64, ans)
0x43f0000000000000

It seems to me that this should either throw an error, or try to find a numeric type that the integer fits into (perhaps just default to BigInt although that would be rather inefficient).

Provide pure lazy functionality for handling/querying extremely large files

Currently, the generic read function requires allocating an internal tape large enough to hold the entire json object being parsed; for extremely large objects, this is problematic because we can't allocate an internal tape large enough. We could, however, provide LazyJSON-like functionality by allowing a lazy access to the parsed JSON object/array.

Struct map a `JSON3.Object` to a struct?

Suppose that I read some JSON in like this:

json_object = JSON3.read(my_json_string) # `json_object` is of type `JSON3.Object`

Can I now convert json_object to a struct of type T, where I have done the necessary work to set up struct mapping for T?

Obviously, I can do this with:

json_object = JSON3.read(my_json_string) # `json_object` is of type `JSON3.Object`
temp_string = JSON3.write(json_object) # `temp_string` is a `String
my_object = JSON3.read(temp_string, T) # now `my_object` is of type `T`

But is there a way to do this that doesn't require me to write it out to an intermediate string? I guess I'm looking for a method struct_map such that this works:

json_object = JSON3.read(my_json_string)
my_object = JSON3.struct_map(json_object, T) # I want `my_object` to be of type `T`

Could have a specialized read for a concrete NamedTuple type like we have for concrete Tuple type

Roundtrip is broken for Floats

Hello,

I observed this behaviour in your library and wanted to let you know. I am actually relying on a library that uses JSON2, which suffers from similar behaviour and it's causing some serialization issues for me. We were discussing the possibility of just changing the dependency to JSON3, but the problem persists.

using JSON
using JSON3

JSON.parse(JSON.json(2.0)) # Float -> Float
JSON3.read(JSON3.write(2.0)) # Float -> Int

# Dict. Although This might be intended?
JSON.parse(JSON.json(Dict("abc" => 1))) # Dict -> Dict
JSON3.read(JSON3.write(Dict("abc" => 1))) # Dict -> Object

I saw something similar reported in the JSON2 repository.

Thank you for your time.

Add support for parsing one object per line

Right now there is no option to read a file that is structured like this.

{"a": 1}
{"a": 3}

BigQuery and Apache Spark output their json in this format.

Pandas has a neat feature pd.read_json(..., lines=True), which allows it to parse files that are structured like this.

This speedbump has prevented me from reading a JSON file into a DataFrame for a few days now.

PS: I'm open to contributing if I could get some pointers on where to start

JSON-encoding a Symbol may result in invalid JSON

This happens because the value of a symbol is not escaped and e.g. double quotes in the input turn into unesaped double quotes in the output:

julia> JSON3.read(JSON3.write([Symbol("before \" after")]))
ERROR: ArgumentError: invalid JSON at byte position 12 while parsing type JSON3.Array: ExpectedComma

This can happen as a result of reading a JSON file:

julia> print(copy(JSON3.read("""{"before \\" after": true}""")))
Dict{Symbol,Any}(Symbol("before \" after") => true)

Fail to read large nested data with mutable struct

Hi there,

I have a Dict that contains some large vectors, around 6M elements each. FYI the code is here. Most of these vectors have eltype Int or Vector{Int}. 2 of the vectors contain custom mutable structs. After enabling constructors taking no args and setting the following:

JSON3.StructTypes.StructType(::Type{Person{A, S}}) where {A, S} = JSON3.StructTypes.Mutable()
JSON3.StructTypes.StructType(::Type{households.Household})      = JSON3.StructTypes.Mutable()

I can then write to disk as follows:

s = JSON3.write(d)
write(filename, s)

So far so good. The file is around 1.5GB.

I run into trouble reading back from disk...

s = String(read(filename))  # Ok. Takes about 1 second.
d = JSON3.read(s)  # Error

The error is:

ERROR: SystemError: mmap: The paging file is too small for this operation to complete.
Stacktrace:
 [1] windowserror(::Symbol, ::UInt32; extrainfo::Nothing) at .\error.jl:183
 [2] #windowserror#52 at .\error.jl:182 [inlined]
 [3] windowserror at .\error.jl:182 [inlined]
 [4] mmap(::Mmap.Anonymous, ::Type{Array{UInt64,1}}, ::Tuple{Int64}, ::Int64; grow::Bool, shared::Bool) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Mmap\src\Mmap.jl:218
 [5] #mmap#14 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Mmap\src\Mmap.jl:251 [inlined]
 [6] mmap at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Mmap\src\Mmap.jl:251 [inlined]
 [7] read(::String) at C:\Users\jock\.julia\packages\JSON3\pjUse\src\read.jl:23
 [8] top-level scope at REPL[16]:1

I'm fairly confident this is user error and that I'm not leveraging the full power of JSON3.
Any tips?

higher rank arrays should appear as nested lists

For example, see the behavior of JSON.jl

julia> A = [1 0
            0 1]
2×2 Array{Int64,2}:
 1  0
 0  1

julia> JSON.json(A)
"[[1,0],[0,1]]"

julia> A = rand(2,2,2)
2×2×2 Array{Float64,3}:
[:, :, 1] =
 0.293369   0.187349
 0.0572265  0.719968

[:, :, 2] =
 0.521351  0.454227
 0.546091  0.773648

julia> JSON.json(A)
"[[[0.2933693401153792,0.057226458190521745],[0.18734882624283,0.7199679429526]],[[0.52135120453497,0.54609145253139],[0.4542265690601863,0.7736478960773061]]]"

It would also be nice to at least have the option for there to be an inverse of this. That is, a function which reads in a nested JSON and automatically constructs an AbstractArray of the appropriate rank. For me personally, this would be a great default behavior in cases where all elements are a common type, but I can see how this would be controversial.

It might also be worth considering doing column-wise instead of row-wise ordering because that would match Julia's memory layout, however this might clash with conventions that exist outside of Julia.

Slight incompatibility between LazyJSON.Object and JSON3.write

Hi! Thanks for JSON3 (and JSON2)!

julia> using LazyJSON, JSON3

julia> object = LazyJSON.value("""{"foo":"bar"}""")
LazyJSON.Object{Nothing,String} with 1 entry:
  "foo" => "bar"

julia> JSON3.write(object) |> println
{"\"foo\"":"bar"}

julia> JSON3.write(Dict("foo" => "bar")) |> println
{"foo":"bar"}

It's awesome that this even works at all, but as you can see, LazyJSON.Object keys somehow get extra quotes when printed. Since compatibility is already 99% there, it would be nice if this issue could be fixed. I don't know if the problem is on your end or LazyJSON's end, but I opened an issue over there as well.

Code generation based on JSON schemas

It would be great to automatically generate parsing code based on a JSON schema like in https://github.com/quicktype/quicktype. I made an attempt at parsing JSON schemas here, but wasn't quite successful yet. See the discussion here: fredo-dedup/JSONSchema.jl#6

The need for a type::String field and subtypekey for abstract types

Adding an additional type::String field to all my concrete types – a field that will always contain the name of said concrete type – is irritating, for lack of a better word. I think I understand its purpose, but it would be amazing to avoid the need of adding this "type" field to all our concrete types.

For me this becomes relevant when I need to save a vector that contains elements that can be of any of the sub (concrete) types of a single abstract type. Currently, I just use a Union of all the possible concrete types:

v = JSON3.read(io, Vector{Union{subtypes(MyAbstractType)...}})

and then if I must I can convert that vector of unions to a Vector{MyAbstractType}:

convert(Vector{MyAbstractType}, v)

This seems like a preferable workaround at least for my use-case, but I would love to hear what you think about this "issue" (it is after all more of a gripe).

How to allow read on ObjectTypes with non-String/Symbol key type

Segmentation faults

As I'm really excited about this new stab at JSON parsing, I gave it a try right away and I bumped into several segfaults (julia crashing).
As they were unpredictable, this is the closest to a reproducible example I could get.
This works:

using JSON3
txt = """
{ "a" : { "b" : [ 1, 2 ],
          "c" : [ 3, 4 ] } }
"""

jj = JSON3.read(txt)
jj.a.b

This, however, when pasted in a fresh Julia session, most likely gives the wrong result or, more often than not, brings down the session:

using JSON3
txt = """
{ "a" : { "b" : [ 1, 2 ],
          "c" : [ 3, 4 ] } }
"""

JSON3.read(txt).a.b

This happens only the first time the last line is executed in a new session: i.e. subsequent executions work as expected.
Looks like some uninitialized memory access...
I'm on Julia 1.1.1

`car` example fails? v1.3.0 (in README)

the example in the test file works, the one in the README did not

julia> using JSON3
[ Info: Precompiling JSON3 [0f8b85d8-7281-11e9-16c2-39a750bddbf1]

julia> abstract type Vehicle end

julia>

julia> struct Car <: Vehicle
           type::String
           make::String
           model::String
           seatingCapacity::Int
           topSpeed::Float64
       end

julia>

julia> struct Truck <: Vehicle
           type::String
           make::String
           model::String
           payloadCapacity::Float64
       end

julia>

julia> JSON3.StructType(::Type{Vehicle}) = JSON3.AbstractType()

julia> JSON3.subtypekey(::Type{Vehicle}) = :type

julia> JSON3.subtypes(::Type{Vehicle}) = (car=Car, truck=Truck)

julia>

julia> car = JSON3.read("""
       {
           "type": "car",
           "make": "Mercedes-Benz",
           "model": "S500",
           "seatingCapacity": 5,
           "topSpeed": 250.1
       }""", Vehicle)
ERROR: ArgumentError: Car doesn't have a defined `JSON3.StructType`
Stacktrace:
 [1] read(::JSON3.NoStructType, ::Base.CodeUnits{UInt8,String}, ::Int64, ::Int64, ::UInt8, ::Type{Car}) at C:\Users\jas\.julia\packages\JSON3\PVBst\src\structs.jl:301
 [2] read(::String, ::Type{Vehicle}) at C:\Users\jas\.julia\packages\JSON3\PVBst\src\structs.jl:920
 [3] top-level scope at REPL[24]:1

Constructing immutable structs from keyword constructors

I bumped into this today and think it would also help with #69

julia> Base.@kwdef struct Test
           t1::String=""
           t2::String=""
       end
Test

julia> StructType(::Type{Test}) = Struct()
StructType

julia> JSON3.read("""{"t1":"Hello","t2":"world"}""",Test)
Test("Hello", "world")

julia> JSON3.read("""{"t1":"Hello"}""",Test)
ERROR: ArgumentError: invalid JSON at byte position 16 while parsing type String: UnexpectedEOF
{"t1":"Hello"}

julia> JSON3.read("""{"t2":"Hello","t1":"world"}""",Test)
Test("Hello", "world")

The last one (related to #69 ) threw me 😅

It would be awesome if JSON3 could somehow try to use a keyword constructor so

julia> JSON3.read("""{"t1":"Hello"}""",Test)

maps to

Test(t1="Hello")

Less verbose pretty()

Great addition the new pretty printing feature! 👍

Is is possible to mute the str = "..." messages?
It looks like they have mainly a debugging purpose.

julia> JSON3.pretty(JSON3.write([1,2]))
str = "[1,2]"
[
  str = "1"
1,
  str = "2"
2
]

32 bit not working

First off, great package! I just tried switching GeoJSON.jl over to this use package (PR). But I noticed AppVeyor 32 bit failed with bitcast: argument size does not match size of target type.

I see there is an appveyor.yml, but the service doesn't seem like it is turned on. It would help to catch 32 bit errors. In this case this line fails:

JSON3.jl/src/utils.jl

Line 71 in 37ab02e

string(len) = STRING | Core.bitcast(UInt64, len)

Since len is an Int i.e. Int32 on 32 bit, which refuses to cast to 64 bits.
The Int comes from here:

JSON3.jl/src/read.jl

Line 195 in 37ab02e

@inbounds tape[tapeidx] = string(keylen)

User provided mapping

Reading the doc I couldn't figure out how to map json keys to object properties where they are different. Unfortunately occassionally it is required with our evolving json config files (which are neither only small nor flat)

struct Foo
    attr_1
    var_2
end

{ "var1": 1, "var_2": 2 }

How can I map var1 onto attr_1 ?

Maintaining 2 structs of everything and then copy/mapping them is awkward. May be if users could provide a mapper ((path, old_key) -> new_key) that gets invoked before assigning the value to an attribute?

thanks a lot for your help.

A proper documentation workflow, with 'stable/latest' links from readme

Currently, README.md in master has already been updated to refer to StructTypes as a separate package. Meanwhile, in the latest tagged release of JSON3 the new package has not yet been factored out. This is causing confusion.

The solution would be to have a very brief generic description in README.md, and have proper documentation built by Documenter (or any other preferred solution), which would follow the usual 'latest/stable' contract, and put the links to both versions on the README.

Slowdown reading Mutable Structs (on master vs 0.1.13)

A benchmark of some local code (that I can't share, unfortunately)

On master:

julia> @btime read_data("test_data.json");
  75.436 ms (838945 allocations: 73.81 MiB)

On v0.1.13:

julia> @btime read_data("test_data.json");
  32.938 ms (365505 allocations: 16.01 MiB)

Profiling shows that the slowdown occurs in the MutableClosure / applyfield! code. Reverting just that section to the v0.1.13 version restores the timing/allocations.

Define JSON3.StructTypes for all the standard libraries

Maybe there is a way to automate this?

For example:

using JSON3, Dates
JSON3.write(now())

potential security issue with big ints

Micheal Griffiths found a tweet about a bug in pythons handling of JSON, which can lead to DOS of python web APIs.

Michael Griffiths(opens in new tab)  4:28 PM
This is a fun one: https://twitter.com/vadimlearning/status/1257743849834897408Seems to work fine in JavaScript (just returns infinity once it overflows, which happens well before that); though it appears to be hanging JSON3.jl for me.

they tried the following in julia and reported that it hangs.

using JSON3
big_number = join(rand(1:9, 100000000), "");
hangs = JSON3.read("{\"number\": $(big_number)}")

Could be an issue.

Pretty printing

Thanks for the lovely package!

One feature that I miss from JSON2.jl is pretty printing. One quick way of providing this would be just lifting pretty.jl from there into this package. Ideally it would probably be neater from the users' point of view to have a keyword argument in write(..., pretty=true), such that we don't need to blow up the string in memory for large structs.

Error when encoding a PNG image as JSON

I'm trying to encode an image as part of a JSON payload, and JSON3 doesn't seem to like it very much. With JSON.jl, it works fine.

julia> using JSON3

julia> s = read("test.png", String);

julia> JSON3.write(s)
ERROR: UndefRefError: access to undefined reference
Stacktrace:
 [1] getindex at ./array.jl:811 [inlined]
 [2] getindex at ./multidimensional.jl:557 [inlined]
 [3] #write#60 at /home/degraafc/.local/share/julia/packages/JSON3/AtAHl/src/write.jl:242 [inlined]
 [4] write at /home/degraafc/.local/share/julia/packages/JSON3/AtAHl/src/write.jl:236 [inlined]
 [5] write(::String; kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/degraafc/.local/share/julia/packages/JSON3/AtAHl/src/write.jl:24
 [6] write(::String) at /home/degraafc/.local/share/julia/packages/JSON3/AtAHl/src/write.jl:22
 [7] top-level scope at REPL[3]:1

Here's the image:

Correct handling of escape \/

\/ escape is not handled correctly I think. E.g.

julia> JSON.parse("{\"a\":\"b\\/c\"}")
Dict{String,Any} with 1 entry:
  "a" => "b/c"

but

julia> JSON3.read("{\"a\":\"b\\/c\"}")
JSON3.Object{Base.CodeUnits{UInt8,String},Array{UInt64,1}} with 1 entryError showing value of type JSON3.Object{Base.CodeUnits{UInt8,String},Array{UInt64,1}}:
ERROR: ArgumentError: encountered invalid escape character in json string: "b\/c"

(it would be great to get it fixed and tagged, in mid August I am running a workshop in which I will use JSON files containing such escapes).

Thank you!

Automatically skip all `nothing`s in a struct?

I have a lot of structs that look like this:

mutable struct Foo
    field_1::Union{Some_Type_1, Nothing}
    field_2::Union{Some_Type_2, Nothing}
    # ...
    field_50::Union{Some_Type_50, Nothing}
end

I want to skip the writing of any field that is nothing. Currently, I can do this by listing out all of those fields and giving it to StructTypes.omitempties. But this gets tedious when I have so many structs that have so many fields.

Is there a more concise way to tell JSON3 to automatically apply omitempties to all fields in a struct?

Mutable struct parsing freezes Julia (after v0.1.10)

Hi,

The code below works well in v0.1.10 but not in v0.1.13 or 1.0: the REPL freezes to the point of ctrl+c not working...

I could not find anything in the documentation to fix it.

Many thanks

Luis

data="""{"latestCandles":[{"instrument":"EUR_USD","granularity":"M1","candles":[{"complete":true,"volume":15,"time":"2020-02-21T21:58:00.000000000Z","bid":{"o":"1.08472","h":"1.08472","l":"1.08447","c":"1.08467"},"ask":{"o":"1.08506","h":"1.08567","l":"1.08497","c":"1.08567"}},{"complete":true,"volume":13,"time":"2020-02-21T21:59:00.000000000Z","bid":{"o":"1.08460","h":"1.08464","l":"1.08427","c":"1.08459"},"ask":{"o":"1.08560","h":"1.08560","l":"1.08488","c":"1.08488"}}]},{"instrument":"EUR_CHF","granularity":"M5","candles":[{"complete":true,"volume":26,"time":"2020-02-21T21:50:00.000000000Z","mid":{"o":"1.06126","h":"1.06130","l":"1.06121","c":"1.06124"}},{"complete":true,"volume":62,"time":"2020-02-21T21:55:00.000000000Z","mid":{"o":"1.06126","h":"1.06155","l":"1.06126","c":"1.06128"}}]}]}"""

mutable struct candlestickdata
    o   #open
    h   #high
    l   #low
    c   #close

    candlestickdata() = new()
end

mutable struct candlestick
    complete
    volume
    time
    bid::candlestickdata
    ask::candlestickdata
    mid::candlestickdata

    candlestick() = new()
end

mutable struct candles
    instrument::String
    granularity::String
    candles::Vector{candlestick}

    candles() = new()
end

mutable struct latestCandles
    latestCandles::Vector{candles}

    latestCandles() = new()
end

# Declaring JSON3 struct types
JSON3.StructType(::Type{candlestickdata}) = JSON3.Mutable()
JSON3.StructType(::Type{candlestick}) = JSON3.Mutable()
JSON3.StructType(::Type{candles}) = JSON3.Mutable()
JSON3.StructType(::Type{latestCandles}) = JSON3.Mutable()

JSON3.read(data,latestCandles)

'''

Dict with UUID keys

I'm trying to write and read a Dict with UUID keys. I'm not sure what I'm doing wrong, I tried the various definitions of StructType but this doesn't work:

using JSON3, UUIDs
d = Dict(uuid1() => i for i in 1:3)
JSON3.StructType(::Type{UUID}) = JSON3.Struct() # maybe I'm not defining the right one here?
t = JSON3.write(d)
d1 = JSON3.read(t, Dict{UUID, Int}) # ERROR: MethodError: no method matching keyvalue(::Type{UUID}, ::Bool, ::Ptr{UInt8}, ::Int64)

integration with CategoricalArrays.jl

This happens:

┌ Warning: `ncodeunits(x::CategoricalString)` is deprecated, use `ncodeunits(String(x))` instead.
│   caller = write(::JSON3.StringType, ::Array{UInt8,1}, ::Int64, ::Int64, ::CategoricalString{UInt8}) at write.jl:251
└ @ JSON3 ~\.julia\packages\JSON3\phpk5\src\write.jl:251
┌ Warning: `ncodeunits(x::CategoricalString)` is deprecated, use `ncodeunits(String(x))` instead.
│   caller = macro expansion at simdloop.jl:69 [inlined]
└ @ Core .\simdloop.jl:69
┌ Warning: `codeunit(x::CategoricalString, i::Integer)` is deprecated, use `codeunit(String(x), i)` instead.
│   caller = macro expansion at write.jl:243 [inlined]
└ @ Core ~\.julia\packages\JSON3\phpk5\src\write.jl:243
┌ Warning: `codeunit(x::CategoricalString, i::Integer)` is deprecated, use `codeunit(String(x), i)` instead.
│   caller = macro expansion at write.jl:265 [inlined]
└ @ Core ~\.julia\packages\JSON3\phpk5\src\write.jl:265

when we write a DataFrame containing categorical strings.

@nalimilan - do we have to change this or with the next release of CategorcialArrays.jl this will be gone?

CC @quinnj

Discussion: relax order requirement for StructTypes.Struct()

This isn't a bug, as it is well documented, but I wanted to raise the possibility of changing this behavior. Possibly related to #47 Simple example

using JSON3
using StructTypes

struct Foo
    a::Int64
    b::Int64
end
StructTypes.StructType(::Type{Foo}) = StructTypes.Struct()

julia> JSON3.read("""{"b": 1, "a": 2}""", Foo).a
1 # oops!

The two issues I see are

• (in my opinion) - breaks principle of least surprise. Using the "default" julia type (struct) with a JSON parser (unordered), implicitly requires ordered data
• Forces uses of mutable structs to safely interact with JSON3

That all said I don't know how technically feasible this is - naively I would think because you know the field order of Foo, the parser could re-order data as it receives it to match, but assume this would at minimum cost some performance.

Relaxed JSON

thanks a lot for this great package.

We have plenty json config files, which are maintained by people. Does JSON3 support some sort of relaxed JSON syntax, e.g.

• support comments, such as '#' or '//'
• { a: 1 } instead of {"a": 1 }
• { a: 1, } ignore extra comma
• {
  a: 1
  b: 2,
  c: 3
  } ',' may be skipped in multi-line constructs outside multi-line string "..\r\n..". Yet ',' may optionally still be used.
• may be even: optional { } on root level

If not supported yet, would it be complicated to add this feature? May be via read_relaxed(..) or similar.

Confused with the nested `JSON3.construct`

julia> using JSON3

julia> struct A
             b::Int
             c::Int
             end

julia> struct B
              a::A
              b::Int
              end

julia> JSON3.StructType(::Type{B}) = JSON3.ObjectType()

julia> JSON3.StructType(::Type{A}) = JSON3.ObjectType()

julia> JSON3.construct(::Type{B}, x::Dict) = B(JSON3.construct(A, x[:a]), x[:b])

julia> JSON3.construct(::Type{A}, x::Dict) = A(x[:b], x[:c])

julia> JSON3.read("""
              {
                  "b":1,
                  "a": {"b":5, "c":8 }
              }""", B)
ERROR: KeyError: key :b not found
......

But if I define the constructor of A like this:

julia> JSON3.construct(::Type{A}, x::Dict) = A(x["b"], x["c"])

julia> JSON3.read("""
              {
                  "b":1,
                  "a": {"b":5, "c":8 }
              }""", B)
B(A(5, 8), 1)

It works fine. However, the following will not work now:

julia> JSON3.read("""
       {
           "b":1,
           "c":2
       }""", A)
ERROR: KeyError: key "b" not found
Stacktrace:
......

Any suggestions for how to deserialize nested struct?

Reading empty arrays

julia> JSON3.read("[]")
0-element JSON3.Array{Union{},Base.CodeUnits{UInt8,String},Array{UInt64,1}}

So for this simple example I can write JSON3.read("[]",Vector{String}) but how can I tell that in a case like:

julia> JSON3.read("{\"paths\":[]}")
JSON3.Object{Base.CodeUnits{UInt8,String},Array{UInt64,1}} with 1 entry:
  :paths => Union{}[]

?

Extreme compile times for read(data, Vector{CustomType})

Trying to use JSON3 instead of JSON for https://github.com/fredrikekre/Canvas.jl and found that there are very long compile times in some cases. I tried to reduce it to just this file: https://gist.github.com/fredrikekre/f3f52938656c610b839c24033d54ba89 with the following results:

$ julia -e 'include("mwe.jl"); @time JSON3.read(array_data, Vector{Course}); @time JSON3.read(array_data, Vector{Course})'
 35.656557 seconds (66.28 M allocations: 4.340 GiB, 4.40% gc time)  <---- ??
  0.000111 seconds (194 allocations: 10.219 KiB)

Reading to JSON3.Array is fast, reading the same data (modulo []) to JSON3.Object is fast, and reading to CustomType is at least reasonably fast:

$ julia -e 'include("mwe.jl"); @time JSON3.read(array_data); @time JSON3.read(array_data)'
  0.936212 seconds (3.42 M allocations: 172.945 MiB, 8.15% gc time)
  0.000039 seconds (13 allocations: 7.969 KiB)

$ julia -e 'include("mwe.jl"); @time JSON3.read(object_data); @time JSON3.read(object_data)'
  0.935497 seconds (3.42 M allocations: 172.813 MiB, 8.37% gc time)
  0.000013 seconds (8 allocations: 7.719 KiB)

$ julia -e 'include("mwe.jl"); @time JSON3.read(object_data, Course); @time JSON3.read(object_data, Course)'
  5.611611 seconds (15.75 M allocations: 947.468 MiB, 9.85% gc time)
  0.000100 seconds (184 allocations: 9.875 KiB)

The array_data is just of length 1 with the same object inside. Is there a better way to read data where you want Vector{CustomType} back? Maybe just use JSON3.read() and construct the vector myself from the resulting JSON3.Array?

Anyway, I thought reading something to Vector{CustomType} is common enought that it might be interesting to optimize or add special methods for.

show for JSON3.Array

It would be great if JSON3.Array had the same show method as normal array. See:

julia> show(JSON3.read(JSON3.write([1,2,3])))
[
  1,
  2,
  3
]
julia> show(copy(JSON3.read(JSON3.write([1,2,3]))))
[1, 2, 3]

And the problem is that:

• it is visually verbose
• it corrupts display if such an object is stored in e.g. data frame (which expects show to produce a single line output for an array

JSON3.pretty is not compatible with allow_inf

It seems that pretty is not compatible with allow_inf...
JSON3.pretty( JSON3.write(Dict( "x" => Inf64), allow_inf=true) )

{
ERROR: ArgumentError: invalid JSON at byte position 6 while parsing type Any: InvalidChar
{"x":Inf}

Stacktrace:
 [1] invalid(::JSON3.Error, ::Base.CodeUnits{UInt8,String}, ::Int64, ::Type{T} where T) at C:\Users\SteffenPL\.julia\packages\JSON3\QAppa\src\JSON3.jl:24
 [2] read(::StructTypes.Struct, ::Base.CodeUnits{UInt8,String}, ::Int64, ::Int64, ::UInt8, ::Type{Any}; kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\SteffenPL\.julia\packages\JSON3\QAppa\src\structs.jl:67
 [3] read at C:\Users\SteffenPL\.julia\packages\JSON3\QAppa\src\structs.jl:43 [inlined]
 [4] pretty(::Base.TTY, ::String, ::Int64, ::Int64; kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\SteffenPL\.julia\packages\JSON3\QAppa\src\pretty.jl:42
 [5] pretty at C:\Users\SteffenPL\.julia\packages\JSON3\QAppa\src\pretty.jl:8 [inlined] (repeats 2 times)
 [6] pretty(::String; kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\SteffenPL\.julia\packages\JSON3\QAppa\src\pretty.jl:5        
 [7] pretty(::String) at C:\Users\SteffenPL\.julia\packages\JSON3\QAppa\src\pretty.jl:5
 [8] top-level scope at none:1

Thanks for the great package! 👍

JSON3.write expects the output of pairs to have length defined

The pairs function from Base is only required to return an iterator, but if you define e.g. JSON3.StructType(::Type{MyType}) = JSON3.ObjectType() and Base.pairs(x::MyType) = Iterators.filter(isnothing∘last, ("foo" => "bar", "foobar" => nothing)), JSON3.write throws a MethodError:

MethodError: no method matching length(::Base.Iterators.Filter{Base.var"#56#57"{Base.var"#58#59"{typeof(isnothing)},typeof(last)},Base.Generator{NTuple{6,Symbol},MatrixProtocolClient.var"#11#12"{MatrixProtocolClient.LoginBody{MatrixProtocolClient.UserIdentifier}}}})
Closest candidates are:
  length(!Matched::Core.SimpleVector) at essentials.jl:593
  length(!Matched::Base.MethodList) at reflection.jl:849
  length(!Matched::Core.MethodTable) at reflection.jl:923
  ...

Stacktrace:
 [1] #write#68(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(JSON3.write), ::JSON3.ObjectType, ::Array{UInt8,1}, ::Int64, ::Int64, ::MatrixProtocolClient.LoginBody{MatrixProtocolClient.UserIdentifier}) at /home/adam/.julia/packages/JSON3/ItGdr/src/write.jl:129
 [2] write(::JSON3.ObjectType, ::Array{UInt8,1}, ::Int64, ::Int64, ::MatrixProtocolClient.LoginBody{MatrixProtocolClient.UserIdentifier}) at /home/adam/.julia/packages/JSON3/ItGdr/src/write.jl:127
 [3] #write#55(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(JSON3.write), ::MatrixProtocolClient.LoginBody{MatrixProtocolClient.UserIdentifier}) at /home/adam/.julia/packages/JSON3/ItGdr/src/write.jl:24
 [4] write(::MatrixProtocolClient.LoginBody{MatrixProtocolClient.UserIdentifier}) at /home/adam/.julia/packages/JSON3/ItGdr/src/write.jl:22
 [5] top-level scope at In[26]:1

JSON3 v0.1.13

Parsing before construct

This is a question/feature request more than a issue. If this is not the right place to post let me know (maybe discourse...)

I am helping in a package that uses intensively JSON3 for parsing market data. Everything works perfect but the json source data has float numbers and dates as strings so we have to declare struct with no types (which I've read is bad for performance) and then change the field to the appopriate type (also bad).

Is there a way to avoid this and parse the data into the appropriate type before the field is updated (because it is #undef at that moment)? If not, could it be implemented?

Maybe the solution is in StructType.construct and StructType.keywords but I do not know how to make it work.

Thank you.

Luis

Here is an example of the code:

• Struct used for price data (with no types)

   mutable struct priceBucket
       price # float as string
       liquidity 

      priceBucket() = new()
   end

    mutable struct price
       type 
       instrument
       time # Date as string
       bids::Vector{priceBucket}
       asks::Vector{priceBucket} 
       closeoutBid # float as string
       closeoutAsk # float as string
       tradeable 

       price() = new()
   end

   JSON3.StructType(::Type{priceBucket}) = JSON3.Mutable()
   JSON3.StructType(::Type{price}) = JSON3.Mutable()

• Function we have to use to transform to the right type after the struct is populated

function coercePrice(price::price)`
    # Coerce Asks
    for ask in price.asks
        ask.price = parse(Float32, ask.price)
    end
    # Coerce Bids
    for bid in price.bids
        bid.price = parse(Float32, bid.price)
    end
    price.time = DateTime(first(price.time, 23), Dates.DateFormat("yyyy-mm-ddTHH:MM:SS.sssssssssZ"))
    price.closeoutBid = parse(Float32, price.closeoutBid)
    price.closeoutAsk = parse(Float32, price.closeoutAsk)

    return price
    end

Support Tuple-type parsing for individual elements

Reading a JSON encoded Symbol (which encodes as a string) requires Meta.parse rather than parse.

this is from see https://discourse.julialang.org/t/how-to-read-tuples-with-json3/29843

Issue with Array eltype promotions

When parsing arrays, the eltype looks correct here:

JSON3.read("[1.2, 2.0]")
2-element JSON3.Array{Union{Float64, Int64}, ...

but not here:

JSON3.read("[1.2, 2.0, 3.3]")
3-element JSON3.Array{Any, ...

I would have expected Union{Float64, Int64}, or even better, just Float64 in both cases.

Read a float in as a String?

Suppose I do this: x = JSON3.read("1.200", String)

Expected behavior:

1. The type of x is String.
2. The value of x is "1.200"

Actual behavior:

julia> import JSON3

julia> x = JSON3.read("1.200", String)
ERROR: ArgumentError: invalid JSON at byte position 1 while parsing type String: ExpectedOpeningQuoteChar
1.200

Stacktrace:
 [1] invalid(error::JSON3.Error, buf::Base.CodeUnits{UInt8, String}, pos::Int64, T::Type)
   @ JSON3 ~/.julia/packages/JSON3/vUxOS/src/JSON3.jl:24
 [2] read(::StructTypes.StringType, buf::Base.CodeUnits{UInt8, String}, pos::Int64, len::Int64, b::UInt8, ::Type{String}; kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ JSON3 ~/.julia/packages/JSON3/vUxOS/src/structs.jl:109
 [3] read
   @ ~/.julia/packages/JSON3/vUxOS/src/structs.jl:82 [inlined]
 [4] #read#15
   @ ~/.julia/packages/JSON3/vUxOS/src/structs.jl:24 [inlined]
 [5] read(str::String, ::Type{String})
   @ JSON3 ~/.julia/packages/JSON3/vUxOS/src/structs.jl:23
 [6] top-level scope
   @ REPL[1]:1

@quinnj Any idea how I would go about reading a number in as a String instead of a number?

ArgumentError: invalid JSON at byte position 11 while parsing type Int64: InvalidChar?

Hi,
Suppose I have two Structs one for Tweet and another for Author of the tweet as follows:

struct User
    id::Union{Int64, Nothing}
    id_str::Union{String, Nothing}
    created_at::Union{String, Nothing}
    name::Union{String, Nothing}
    screen_name::Union{String, Nothing}
    location::Union{String, Nothing}
    statuses_count::Union{Int64, Nothing}
    followers_count::Union{Int64, Nothing}
    description::Union{String, Nothing}
    profile_image_url::Union{String, Nothing}
end

struct Tweet
    id::Union{Int64, Nothing}
    id_str::Union{String, Nothing}
    created_at::Union{String, Nothing}
    favorite_count::Union{Int, Nothing}
    retweet_count::Union{Int, Nothing}
    full_text::Union{String, Nothing}
    lang::Union{String, Nothing}
    place::Union{String, Nothing}
    truncated::Union{Bool, Nothing}
    user::User
end

Now I have a Json String as :

json_str = "{\"id_str\":\"1305501948074835974\",\"created_at\":\"Mon Sep 14 13:41:34 +0000 2020\",\"place\":null,\"id\":1305501948074835974,\"user\":{\"name\":\"Donald J. Trump\",\"id_str\":\"25073877\",\"created_at\":\"Wed Mar 18 13:46:38 +0000 2009\",\"id\":25073877}}"

I am trying to define a New Struct function and read the string in to value of type:

JSON3.StructType(::Type{Tweet}) = JSON3.Struct()
JSON3.read(json_str, Tweet)

I am getting the following error:

ArgumentError: invalid JSON at byte position 11 while parsing type Int64: InvalidChar
{"id_str":"1305501948074835974","cre

Does the string need to match the exact order of the Struct? or Not sure where is the Issue. Though the JSON3.read(json_str) works fine.

If someone can tell me:

• How to read such data in to some value of Given struct type (Tweet)
• Is it possible to convert the value to Dictionary using Json3

Constructing fully immutable objects iteratively

I've been bothered by the fact that, for iterative construction of objects, JSON3 requires a mutable struct definition (when deserializing a type).

For fully immutable types (i.e., those which do not contain references outside to other objects, don't allow nothing or missing values, etc), at least, I'm wondering if there is some workaround.

For example, if mutable structs and structs were guaranteed to have the same memory layout, could one construct a mutable struct, and then reinterpret it as an immutable one? (I'm not sure.)

One workaround (for fully immutable types) that I believe would work (see VideoIO.jl/src/util.jl, which modifies fields of structs allocated by ffmpeg libraries):

1. Use introspection to determine the layout of (immutable) struct T using fieldoffset, fieldname, and fieldtype.
2. Allocate a Vector{T}(undef, 1)
3. Reinterpret as a Vector{UInt8}
4. Fill in field values using pointers + unsafe_wrap

Pseudocode might look something like this

obj = Vector{T}(undef, 1)  # should guarantee that the object is in memory; may want to zero the memory
ptr = convert(Ptr{UInt8}, pointer(obj))

# later

key = parsekey()  # parse key from JSON
offset, type = get_offset_type(T, key)

field_ptr = convert(Ptr{type}, ptr + offset)
field_arr = unsafe_wrap(Array, field_ptr, 1)
field_arr[1] = parsetype(type) # parse value from JSON

Obviously, the part which sets the value should be wrapped in a function, and there would probably need to be some way to make it type stable, etc. And it could be brittle. But it might work... :-)

(As a broader goal, I think it would be useful to have an extendable serialization/deserialization infrastructure similar to serde.rs, which allowed easy serialization and deserialization of immutable structs.)

The need to define a StructType for each parametric type

I might be wrong but it seems like we need to define a StructType for each and every parameterized parametric type?

using JSON3
struct MyParametricType{T}
    t::T
    MyParametricType{T}(t) where {T} = new(t)
end
MyParametricType(t::T) where {T} = MyParametricType{T}(t)

x = MyParametricType(1)

JSON3.StructType(::Type{MyParametricType}) = JSON3.Struct()
str = JSON3.write(x) # ERROR: ArgumentError: MyParametricType{String,Int64} doesn't have a defined `JSON3.StructType`

JSON3.StructType(::Type{MyParametricType{Int}}) = JSON3.Struct()
str = JSON3.write(x) # fine