juliadata / structtypes.jl

Abstract definitions and convenience methods for describing, processing, and constructing Julia objects

StructTypes

Package providing the StructTypes.StructType trait for Julia types to declare the kind of "struct" they are, providing serialization/deserialization packages patterns and strategies to automatically construct objects. To learn more, check out the documentation links below.

Installation: at the Julia REPL, import Pkg; Pkg.add("StructTypes")

Maintenance: StructTypes is maintained collectively by the JuliaData collaborators. Responsiveness to pull requests and issues can vary, depending on the availability of key collaborators.

Documentation

structtypes.jl's Issues

Default to `StructTypes.Struct()`?

I wonder if StructType should default to StructTypes.Struct() since that seems to be the most common option. Then probably in more cases JSON serialization will "just work". Though maybe it will be less discoverable how to customize it since you won't get the ArgumentError: X doesn't have a defined StructTypes.StructType` error.

missing docstring warning

https://juliadata.github.io/StructTypes.jl/stable/#StructTypes.OrderedStruct

Macro for subtypes lowering with custom type field

According to this solution for lowering subtypes that does not have additional type field:
The need for a type::String field and subtypekey for abstract types

I've tried to write such a macro, but I think I get this wrong, can you look at it? @quinnj

macro lowersubtype(type, subtypekey)
    T = Core.eval(__module__, type)
    vars = [QuoteNode(e) for e in fieldnames(T)]
    xvars = [:(x.$(e)) for e in fieldnames(T)]
    types = [esc(e) for e in fieldtypes(T)]
    nt = :(NamedTuple{(QuoteNode($subtypekey), $(vars...),), Tuple{String, $(types...)}})
    return quote
        StructTypes.StructType(::Type{$(esc(type))}) = StructTypes.CustomStruct()
        StructTypes.lower(x::$(esc(type))) = (cmd = $subtypekey, $(xvars...))
        StructTypes.lowertype(::Type{$(esc(type))}) = $nt
        $(esc(type))(x::$nt) = $(esc(type))($(xvars...))
    end
end

struct Foo
    x::Int
    y::Float64
end

@lowersubtype Foo "type"

Recommendations how to add subtypes later

Assuming i have a have piece of code which has an StructTypes.AbstractType() which already has it's StructTypes.subtypes(::Type{Vehicle}) set but i now want to add a new subtype at run time.

Is there a recommended way to do that? If so i would offer to extend the documentation with an example. My current solution would be StructTypes.subtypes(::Type{Vehicle}) = merge(StructTypes.subtypes(::Type{Vehicle}), (; train=Train)) but if there is a different way preferred with the package then i would like to know.

Example missing in the documentation

I am not able to create a struct type.

The following code does not work:

using YAML, StructTypes

 mutable struct Test
    can_log::String
    sys_log::String
    version::String
    info::String
end

data = YAML.load_file("data/config.yaml")
test = data["tests"][1]
println(test)

StructTypes.StructType(Test) = StructTypes.Mutable()
StructTypes.constructfrom(Test, test)

Content of the input file:

tests:
    - can_log:   data/27-08-2022/logfile.csv
      sys_log:   data/27-08-2022/syslog
      version:   1.7.1
      info:      first test
    - can_log:   data/28-08-2022/logfile.csv
      sys_log:   data/28-08-2022/syslog
      version:   1.7.2
      info:      second test

StructTypes encourage piracy

Suppose I have a package, in which I need to serialize some data that contain a LongDNA{4}, but do not want to expose the private (non-documented) memory layout of LongDNA. In that case, I'd need to overwrite StructTypes.StructType(::Type{LongDNA{4}}) - or, alternatively, overwrite some JSON3 methods.

Both are type piracy, which can have rather bad consequences. For example, it's unknown to me if the user installs another package which also need to serialize LongDNA, but may choose to do it differently. In that case, my code may randomly malfunction.

I don't see a non-breaking way of getting around it, but perhaps it's worth considering for a potential breaking release in the future:

• This package defines a struct AbstractEncoder end
• Struct mapping is done by StructTypes.StructType(::AbstractEncoder, ::Type{MyType})
• Default struct mapping can use ::AbstractEncoder directly.
• Users can then subtype AbstractEncoder, and create struct mapping using their own subtype.

fails to read integer arrays

fails to read back integer array that it serialized:

julia> using JSON3, StructTypes

julia> struct Struct1
           iarr::Vector{Integer}
       end

julia> StructTypes.StructType(::Type{Struct1}) = StructTypes.Struct()

julia> s1 = Struct1([1,2,3,4,5]);

julia> iob = IOBuffer();

julia> JSON3.write(iob, s1);

julia> s1_json = String(take!(iob))
"{\"iarr\":[1,2,3,4,5]}"

julia> s2 = JSON3.read(s1_json, Struct1)
ERROR: ArgumentError: invalid JSON at byte position 10 while parsing type Integer: ExpectedOpeningObjectChar
{"iarr":[1,2,3,4,5]}

Stacktrace:
  [1] invalid(error::JSON3.Error, buf::Base.CodeUnits{UInt8, String}, pos::Int64, T::Type)

Default to OrderedStruct for NamedTuples?

Would it be possible to define default StructType for NamedTuples as StructTypes.OrderedStruct()? Or are there any obvious conflicts I'm not seeing? It seems like the problems that might arise because of this are similar to those described in #61, but I am unable to come up with any other StructType for NamedTuples, that might be of use.

sometimes dicts are passed to constructors as `Dict{String,Any}`

I started out thinking this only happened for nested types, but as I set out to make a MWE, I got even more confused. Here it is

using JSON3, StructTypes

struct Inner
    a::Int
end
Inner(dct) = Inner(dct[:a])

StructTypes.StructType(::Type{Inner}) = StructTypes.DictType()

struct Outer
    I::Inner
end
Outer(dct) = Outer(dct[:I])

StructTypes.StructType(::Type{Outer}) = StructTypes.DictType()


json = JSON3.write(Dict(:I=>Dict(:a=>1)))
JSON3.read(json, Outer)

gives me

ERROR: LoadError: KeyError: key :I not found
Stacktrace:
 [1] getindex at ./dict.jl:467 [inlined]
 [2] Outer(::Dict{String,Any}) at /home/expandingman/src/scrap.jl:13 (repeats 2 times)
 [3] #construct#2 at /home/expandingman/.julia/packages/StructTypes/CpLmq/src/StructTypes.jl:267 [in
lined]
 [4] construct at /home/expandingman/.julia/packages/StructTypes/CpLmq/src/StructTypes.jl:267 [inlin
ed]
 [5] read(::StructTypes.DictType, ::Base.CodeUnits{UInt8,String}, ::Int64, ::Int64, ::UInt8, ::Type{
Outer}, ::Type{Symbol}, ::Type{Any}; kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),
Tuple{}}}) at /home/expandingman/.julia/packages/JSON3/LtYDK/src/structs.jl:342
 [6] read at /home/expandingman/.julia/packages/JSON3/LtYDK/src/structs.jl:285 [inlined]
 [7] #read#30 at /home/expandingman/.julia/packages/JSON3/LtYDK/src/structs.jl:279 [inlined]
 [8] read at /home/expandingman/.julia/packages/JSON3/LtYDK/src/structs.jl:279 [inlined]
 [9] read(::String, ::Type{Outer}; kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tu
ple{}}}) at /home/expandingman/.julia/packages/JSON3/LtYDK/src/structs.jl:34
 [10] read(::String, ::Type{Outer}) at /home/expandingman/.julia/packages/JSON3/LtYDK/src/structs.jl
:33
 [11] top-level scope at /home/expandingman/src/scrap.jl:19
 [12] include(::String) at ./client.jl:457
 [13] top-level scope at REPL[3]:1
in expression starting at /home/expandingman/src/scrap.jl:19

Usually I would hit this kind of error with Inner where it would try to construct from Dict("a"=>1), I don't know why I suddenly have this problem with the outer one.

As a workaround, it seems that right now one has to define constructors for both Dict{String,Any} and Dict{Symbol,Any} which seems wrong.

Register v1.11.0?

Can we? I'm missing #94 in official version.

`Real` does not deserialize

Julia Version: 1.7.2.

I see that Real is supposed to be StructTypes.NumberType(), but it can't actually deserialize:

julia> JSON3.read("456", Real)
ERROR: ArgumentError: invalid JSON at byte position 1 while parsing type Real: ExpectedOpeningObjectChar
456
julia> JSON3.read("456", Union{Integer, AbstractFloat})
456.0

docs link in README is broken

at https://github.com/JuliaData/StructTypes.jl/blame/master/README.md#L3

DataType is already a thing

typeof(Int) == DataType
As does basically all other types of types, with the exclusion of unions and unionalls.

While we can have multiple things using same name because of shadowing,
I would find a confusing to say this name used for anything other than the original definition

constructfrom having a tough time with a bad Union field

julia> struct MyStruct
           a::Vector{Union{String, Vector{Vector{Vector{Float64}}}}}
       end

julia> using StructTypes

julia> StructTypes.StructType(::Type{MyStruct}) = StructTypes.Struct()

julia> x = Dict(:a=>["z"]);

julia> StructTypes.constructfrom(MyStruct, x)
MyStruct(Union{String, Vector{Vector{Vector{Float64}}}}[[[[122.0]]]])

`StructTypes.x` is not defined

@kshyatt

│┌ @ C:\.julia\packages\JSON3\CpNms\src\structs.jl:32 JSON3.:(var"#read#16")(false, pairs(NamedTuple()::NamedTuple{(), Tuple{}})::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, #self#::typeof(JSON3.read), str::String, _3::Type{TrinityApi._PipeCommunication.Data})
││┌ @ C:\.julia\packages\JSON3\CpNms\src\structs.jl:40  = JSON3.read(JSON3.StructType(T::Type{TrinityApi._PipeCommunication.Data})::StructTypes.UnorderedStruct, buf::Base.CodeUnits{UInt8, String}, pos::Int64, len::Int64, b::UInt8, T::Type{TrinityApi._PipeCommunication.Data})
│││┌ @ C:\.julia\packages\JSON3\CpNms\src\structs.jl:566 JSON3.:(var"#read#50")(pairs(NamedTuple()::NamedTuple{(), Tuple{}})::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, #self#::typeof(JSON3.read), _2::StructTypes.UnorderedStruct, buf::Base.CodeUnits{UInt8, String}, pos::Int64, len::Int64, b::UInt8, _7::Type{TrinityApi._PipeCommunication.Data})
││││┌ @ C:\.julia\packages\JSON3\CpNms\src\structs.jl:617 (JSON3.StructTypes).applyfield::typeof(StructTypes.applyfield)(c, T::Type{TrinityApi._PipeCommunication.Data}, key::Symbol)
│││││┌ @ C:\.julia\packages\StructTypes\AK4aM\src\StructTypes.jl:891 StructTypes.x
││││││ `StructTypes.x` is not defined
│││││└──────────────────────────────────────────────────────────────

add methods for constructing an object directly from an `AbstractDict`

Sometimes I'll have some big dict which I read in in JSON3, and I can't really specify that only some of the elements should be read in as some specific type. I then instead wind up with some nested dicts which have the proper form to be construct into some struct or whatever, but now they are just Julia AbstractDict, and I don't think I have any easy methods to use from StructTypes to get it into the actual struct I want any more.

It would therefore be nice to have something like

construct(dict::AbstractDict, T)

analogous to

JSON3.read(str::AbstractString, T)

Which just builds a struct of type T for me from a dict (likewise from vectors and whatever else).

TagBot is not working

See e.g. https://github.com/JuliaData/StructTypes.jl/runs/1541161385:

Processing version v1.2.0 (75c713f904949c0963bf54246397cab29d7aa63e)
Generating changelog for version v1.2.0 (75c713f904949c0963bf54246397cab29d7aa63e)
Warning: Permanently added the RSA host key for IP address '140.82.114.4' to the list of known hosts.
ERROR: Permission to JuliaData/StructTypes.jl.git denied to deploy key
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Error: TagBot experienced an unexpected internal failure
Traceback (most recent call last):
  File "/root/tagbot/action/__main__.py", line 91, in <module>
    repo.create_release(version, sha)
  File "/root/tagbot/action/repo.py", line 442, in create_release
    self._git.create_tag(version, sha, log)
  File "/root/tagbot/action/git.py", line 103, in create_tag
    self.command("push", "origin", version)
  File "/root/tagbot/action/git.py", line 69, in command
    raise Abort(f"Git command '{cmd}' failed")
tagbot.action.Abort: Git command 'git -C /tmp/tagbot_repo_vex2_4jk push origin v1.2.0' failed

Response (200): {
  "status": "Submitted error report"
}

StructTypes.StructType(::Type{Union{}}) is ambiguous

I was trying to check the StructTypes.StructType of the eltype of some arrays. However, one empty array's eltype was defined as Union{} which threw a method ambiguity error on StructTypes.StructType(::Type{Union{}}).

Typing kinda breaks my brain, so I'm not sure if StructTypes.StructType(::T{}) where T would solve the problem. I also don't know what struct type to use for something that is more nothing than nothing! Maybe this really should throw an error since it doesn't make sense to report the StructType of something that doesn't exist.

Anyway, just thought I'd lift up the weird edge case for your consideration.

read(jsonstring) in docs raises error

https://juliadata.github.io/StructTypes.jl/stable/#StructTypes.AbstractType uses read on a json string

car = StructTypes.read("""
{
    "type": "car",
    "make": "Mercedes-Benz",
    "model": "S500",
    "seatingCapacity": 5,
    "topSpeed": 250.1
}""", Vehicle)

but I think read only works on filenames.

julia> # example from StructTypes deserialization
       car = StructTypes.read("""
       {
           "type": "car",
           "make": "Mercedes-Benz",
           "model": "S500",
           "seatingCapacity": 5,
           "topSpeed": 250.1
       }""", Vehicle)
ERROR: SystemError: opening file "{\n    \"type\": \"car\",\n    \"make\": \"Mercedes-Benz\",\n    \"model\": \"S500\",\n    \"seatingCapacity\": 5,\n    \"topSpeed\": 250.1\n}": No such file or directory

Create (or document) a way to serialize singleton types

[Loosely transcribed from a conversation with @quinnj on Julia Slack]

I'm looking for a way to serialize and unserialize empty singleton structs to JSON, where each singleton is held within another struct, and is also a subtype of some abstract type. Quick example below:

using JSON3, StructTypes
abstract type Super end
struct SubA <: Super end
struct SubB <: Super end

struct Foo{T <: Super}
    x :: T
end

StructTypes.StructType(::Type{<:Foo}) = StructTypes.Struct()
foo = Foo(SubA())

In the above vernacular, I want to be able to serialize an object of type Foo. Here are some of the things I've tried:

1. Doesn't seem to work as StructTypes expects structs to have at least one field

StructTypes.StructType(::Type{<:Super}) = StructTypes.Struct()
JSON3.write(foo)
ERROR: ArgumentError: Cannot access field 1 since type SubA only has 0 fields.

2. Example with super-type from the documentation: https://juliadata.github.io/StructTypes.jl/stable/#AbstractTypes-1

Doesn't work as I don't want my singleton types to have a type field, or similar

3. Define the StructTypes of anything <:Super as StringType. Works for write (sort of), but not deserialization

julia> StructTypes.StructType(::Type{<:Super}) = StructTypes.StringType()
julia> JSON3.write(foo)
"{\"x\":\"SubA()\"}"

julia> JSON3.read(JSON3.write(f), Foo)
ERROR: MethodError: no constructors have been defined for Super

4. Define Super as SuperType and SubA and SubB as Struct

julia> StructTypes.StructType(::Type{Super}) = StructTypes.AbstractType();
julia> StructTypes.StructType(::Type{SubA}) = StructTypes.StringType();
julia> StructTypes.StructType(::Type{SubB}) = StructTypes.StringType();
julia> JSON3.write(foo)
"{\"x\":\"SubA()\"}"

julia> JSON3.read(JSON3.write(foo), Foo)
ERROR: ArgumentError: invalid JSON at byte position 6 while parsing type Super: ExpectedOpeningObjectChar
{"x":"SubA()"}

5. Define Super as StringType and create a Super(s::String) constructor. This one works, though not sure if it's idiomatic.

julia> StructTypes.StructType(::Type{<:Super}) = StructTypes.StringType();
julia> function Super(s::String)
    Dict("SubA()" => SubA(),
         "SubB()" => SubB())[s]
end
julia> JSON3.write(foo)
"{\"x\":\"SubA()\"}"
julia> JSON3.read(ans, Foo)
Foo{SubA}(SubA())

Just asking in general whether 5 seems like a reasonable approach, or if there's some smarter way of doing this that I'm missing? Happy to open a pr with documentation for this once the issue is resolved

updating read for for StructTypes.Mutable()

I have a situation like that:

    Base.@kwdef mutable struct PublisherConfig
        CIName::String = "Test"
        CIFileName::String = "CI.yml"
        SSHSecretName::String = "SSH_private_access"
    end
    JSON3.StructType(::Type{PublisherConfig}) = JSON3.StructTypes.Mutable()
    const CONFIG = PublisherConfig()

Now, the assignment in the last line is important because it shall be a const. the const is important for performance. Thus, what I'm missing is an option to update the pre-existing object from JSON3 (rather than having JSON3 create one)

TODO: Migrate from Travis CI to GitHub Actions CI

Named tuples not converting

Hi, I'm getting this error trying to convert a NamedTuple to store in a NamedTuple field:

ERROR: ArgumentError: type does not have a definite number of fields
Stacktrace:
  [1] fieldcount(t::Any)
    @ Base ./reflection.jl:729
  [2] construct
    @ ~/.julia/packages/StructTypes/Cmlkm/src/StructTypes.jl:573 [inlined]
  [3] constructfrom(#unused#::StructTypes.UnorderedStruct, #unused#::Type{NamedTuple}, #unused#::StructTypes.UnorderedStruct, obj::NamedTuple{(), Tuple{}})
    @ StructTypes ~/.julia/packages/StructTypes/Cmlkm/src/StructTypes.jl:1046
  [4] constructfrom
    @ ~/.julia/packages/StructTypes/Cmlkm/src/StructTypes.jl:1018 [inlined]
  [5] constructfrom(#unused#::Type{NamedTuple}, obj::NamedTuple{(), Tuple{}})
    @ StructTypes ~/.julia/packages/StructTypes/Cmlkm/src/StructTypes.jl:905
  [6] StructClosure
    @ ~/.julia/packages/StructTypes/Cmlkm/src/StructTypes.jl:1041 [inlined]

I made some definitions that seem to fix it, what do you think (seems like both Struct and UnorderedStruct are needed)?

constructfrom(::Struct, ::Type{ST}, ::Struct, obj::OT) where {ST <: NamedTuple, OT <: ST} = obj
constructfrom(::UnorderedStruct, ::Type{ST}, ::UnorderedStruct, obj::OT) where {ST <: NamedTuple, OT <: ST} = obj

Add `DOCUMENTER_KEY` for GitHub Actions

Now that CI has been moved to GitHub Actions, we need to add a new Documenter deploy key.

@quinnj Can you do the following:

Step 1:

julia> import Pkg
julia> Pkg.add("DocumenterTools")
julia> import DocumenterTools
julia> DocumenterTools.genkeys()

Step 2: Go to https://github.com/JuliaData/StructTypes.jl/settings/secrets/actions. Create a new GitHub Secret with name DOCUMENTER_KEY where the value is the Base64-encoded private key from step 1.

Step 3: Go to https://github.com/JuliaData/StructTypes.jl/settings/keys. Create a new deploy key with name DOCUMENTER_KEY_PUBLIC where the value is the public key from step 1. Make sure that the deploy key has write access.

Bug: when calling `applyfield!` on a mutable struct, the 32nd field is skipped

For example, suppose that you have a mutable struct with 40 fields. And you call applyfield! on an instance of this mutable struct. Fields 1 to 31 will be applied correctly. And fields 33 to 40 will be applied correctly. But field number 32 will always be skipped.

This is a big problem, as it means the 32nd field in the struct will always be skipped when e.g. reading JSON in to the struct.

I discovered this bug for the applyfield! function, but it is possible that the bug also exists in the other functions.

Full example:

julia> import StructTypes

julia> mutable struct Foo
           field_1
           field_2
           field_3
           field_4
           field_5
           field_6
           field_7
           field_8
           field_9
           field_10
           field_11
           field_12
           field_13
           field_14
           field_15
           field_16
           field_17
           field_18
           field_19
           field_20
           field_21
           field_22
           field_23
           field_24
           field_25
           field_26
           field_27
           field_28
           field_29
           field_30
           field_31
           field_32
           field_33
           field_34
           field_35
           field_36
           field_37
           field_38
           field_39
           field_40
           Foo() = new()
       end

julia> length(fieldnames(Foo))
40

julia> for fieldname in fieldnames(Foo)
           f_applied = StructTypes.applyfield!((x, y, z) -> nothing, Foo(), fieldname)
           @info "" fieldname f_applied
       end
┌ Info:
│   fieldname = :field_1
└   f_applied = true
┌ Info:
│   fieldname = :field_2
└   f_applied = true
┌ Info:
│   fieldname = :field_3
└   f_applied = true
┌ Info:
│   fieldname = :field_4
└   f_applied = true
┌ Info:
│   fieldname = :field_5
└   f_applied = true
┌ Info:
│   fieldname = :field_6
└   f_applied = true
┌ Info:
│   fieldname = :field_7
└   f_applied = true
┌ Info:
│   fieldname = :field_8
└   f_applied = true
┌ Info:
│   fieldname = :field_9
└   f_applied = true
┌ Info:
│   fieldname = :field_10
└   f_applied = true
┌ Info:
│   fieldname = :field_11
└   f_applied = true
┌ Info:
│   fieldname = :field_12
└   f_applied = true
┌ Info:
│   fieldname = :field_13
└   f_applied = true
┌ Info:
│   fieldname = :field_14
└   f_applied = true
┌ Info:
│   fieldname = :field_15
└   f_applied = true
┌ Info:
│   fieldname = :field_16
└   f_applied = true
┌ Info:
│   fieldname = :field_17
└   f_applied = true
┌ Info:
│   fieldname = :field_18
└   f_applied = true
┌ Info:
│   fieldname = :field_19
└   f_applied = true
┌ Info:
│   fieldname = :field_20
└   f_applied = true
┌ Info:
│   fieldname = :field_21
└   f_applied = true
┌ Info:
│   fieldname = :field_22
└   f_applied = true
┌ Info:
│   fieldname = :field_23
└   f_applied = true
┌ Info:
│   fieldname = :field_24
└   f_applied = true
┌ Info:
│   fieldname = :field_25
└   f_applied = true
┌ Info:
│   fieldname = :field_26
└   f_applied = true
┌ Info:
│   fieldname = :field_27
└   f_applied = true
┌ Info:
│   fieldname = :field_28
└   f_applied = true
┌ Info:
│   fieldname = :field_29
└   f_applied = true
┌ Info:
│   fieldname = :field_30
└   f_applied = true
┌ Info:
│   fieldname = :field_31
└   f_applied = true
┌ Info:
│   fieldname = :field_32
└   f_applied = false
┌ Info:
│   fieldname = :field_33
└   f_applied = true
┌ Info:
│   fieldname = :field_34
└   f_applied = true
┌ Info:
│   fieldname = :field_35
└   f_applied = true
┌ Info:
│   fieldname = :field_36
└   f_applied = true
┌ Info:
│   fieldname = :field_37
└   f_applied = true
┌ Info:
│   fieldname = :field_38
└   f_applied = true
┌ Info:
│   fieldname = :field_39
└   f_applied = true
┌ Info:
│   fieldname = :field_40
└   f_applied = true

JSON order validation for "data types"

I think that leaving key-value order up to the JSON implementation is somewhat brittle and going with the mutable structure is not the right solution.

Instead, it would be better if either

1. the construction would validate that all fields are present and in the right order, eg JSON3.read("{"val2": 2, "val1": 1, "val3": 3}", CoolType) would reject,
2. the construction would order the fields (and also validate).

I would make 2. the default, and 1. optional (for speed). String comparisons should be cheap, and for most applications the extra runtime is worth not working with (silently) messed up data.

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

ArgumentError: invalid JSON at byte position?

Hi, I was trying out the Example of StructType function given in the official document as follows:

struct CoolType
    val1::Int
    val2::Int
    val3::String
end

StructTypes.StructType(::Type{CoolType}) = StructTypes.Struct()

JSON3.read("{"val1": 1, "val2": 2, "val3": 3}", CoolType) 
JSON3.read("{"val2": 2, "val1": 1, "val3": 3}", CoolType) 

First of all compailer complain about "{"val1": 1, "val2": 2, "val3": 3}" being not a valid string and throws error as ERROR: syntax: cannot juxtapose string literal but this one "{\"val2\": 2, \"val1\": 1, \"val3\": \"hey\"}" works. But it looks like the demonstration in the doc string given properly:

Also putting numeric value instead of string in the field JSON3.read("{\"val1\":1,\"val2\":2,\"val3\":3}", CoolType does not work and throws the following error:

ERROR: ArgumentError: invalid JSON at byte position 27 while parsing type String

But the way the documents shows it should be working? @assert JSON3.read("{"val2": 2, "val1": 1, "val3": 3}", CoolType) == CoolType(2, 1, "3")

But the following works fine:

JSON3.read("{\"val2\":2,\"val1\":1,\"val3\":\"hey\"}", CoolType

default value support

Hi, I'm wondering how can I define a field has a corresponding default value, I can't find a trait that defines this property, thus as a result JSON3 will treat field not specified with a value in the payload as Nothing instead of using the default value and fail to create the struct type. I use this type of convention quite frequently in structs created by Configurations.jl, I'm thinking about auto generating the overloads for StructTypes in Configurations. Ideally I'm thinking if the construct method will be able to use the keyword constructor created by things like Configurations.@option or Base.@kwdef so that the default value will be handled automatically

An example of this type of struct looks like the following (part of this is used in Pluto)

@option mutable struct CompilerOptions
    compile::Union{Nothing,String} = nothing
    sysimage::Union{Nothing,String} = nothing
    banner::Union{Nothing,String} = nothing
    optimize::Union{Nothing,Int} = nothing
    math_mode::Union{Nothing,String} = nothing

    # notebook specified options
    # the followings are different from
    # the default julia compiler options

    # we use nothing to represent "@v#.#"
    project::Union{Nothing,String} = "@."
    # we don't load startup file in notebook
    startup_file::Union{Nothing,String} = "no"
    # we don't load history file in notebook
    history_file::Union{Nothing,String} = "no"

    threads::Union{Nothing,String,Int} = get_nthreads()
end

@option struct DistributedOptions
    tunnel::Bool = false
    multiplex::Bool = false
    ssh::Maybe{String} = nothing
    sshflags::Maybe{String} = nothing
    max_parallel::Int = 10
    shell::Symbol = :posix
    dir::String = pwd()
    enable_thread_blas::Bool = false
    exename::String = joinpath(Sys.BINDIR, "julia")
    topology::Symbol = :all_to_all
    lazy::Bool = true
    env::Maybe{Vector{Pair{String, String}}} = nothing
end

@option struct WorkspaceInstance
    name::Symbol = gensym(:workspace)
    uuid::UUID = uuid1()
    compiler::CompilerOptions = CompilerOptions()
    distributed::DistributedOptions = DistributedOptions()
    # scripts that is evaluated in this workspace instance
    linked_scripts::Vector{String} = String[]
end

using CustomStruct lowering for AbstractType

Hello. I'm very new to julia and am trying to use your package in conjunction with JSON3 to do some serialization of an abstract type. Originally I used JSON.jl but stumbled across this package which has the very cool feature of being able to write json to a Type so I am trying to switch over to JSON3.

With JSON.jl, I could specify a lowering function on an abstract type (in this case I am taking in an abstract type and wrapping it in a simple wrapper type called System:

struct System 
    parameters::Vector{Parameter}
    states::Vector{String}
    equations::Vector{String}
    connections::Vector{String}
end

JSON.lower(system::ModelingToolkit.AbstractSystem) = System(
    build_parameters(system),
    string.(states(system)),
    string.(equations(system)),
    get_connections(system)
)

That works fine and nothing else is needed.

With JSON3 however I found that in addition to registering the abstract type as a custom struct, I also have to register the sub types:

StructTypes.StructType(::Type{System}) = StructTypes.Struct()
StructTypes.StructType(::Type{ModelingToolkit.AbstractSystem}) = StructTypes.CustomStruct()
StructTypes.StructType(::Type{ModelingToolkit.ODESystem}) = StructTypes.CustomStruct()


StructTypes.lower(system::ModelingToolkit.AbstractSystem) = System(
    build_parameters(system),
    string.(states(system)),
    string.(equations(system)),
    get_connections(system)
)

if the system passed in to lower is of type ODESystem (or any other subtype of AbstractSystem) then I have to provide that CustomStruct type mapping for that. Is that normal? Am I doing something wrong?

Standard interface for programatic structure definition

I think it would be useful to have a standard interface for programmatic structure. Here I give the basic structure, but StructTypes related features can be added to the functions.

1-time function

functions API:

struct_define(struct_name::Symbol; fields_names::Vector{Symbol}, fields_types::Vector{Union{DataTypes, Symbol, Missing})

Example

struct_define(:Foo, [:a, :b, :c], [:Int64, missing, :String])

over-time method

functions API:

struct_define(struct_data::StructConstructor) # defines a struct based on the data

mutable struct StructConstructor
   name::Symbol
   fields_names::Vector{Symbol} 
   fields_types::Vector{Union{DataType, Symbol, Missing}}
end

# a constructor for when we know the name and number of fields:
function StructConstructor(nfields::Integer)  
   fields_names = Vector{Symbol}(undef, nfields)
   fields_names = Vector{Union{DataType, Symbol, Missing}}(undef, nfields)
   return StructConstructor(:TempName, fields_names, fields_names)
end

function StructConstructor(name::Symbol, nfields::Integer)  
   fields_names = Vector{Symbol}(undef, nfields)
   fields_names = Vector{Union{DataType, Symbol, Missing}}(undef, nfields)
   return StructConstructor(name, fields_names, fields_names)
end

Example

mystruct_data = StructConstructor()
mystruct_data = mystruct_data.name = :Foo # adds name

names = [:a, :b, :c]
types = [:Int64, missing, :String]
for (name, type) in zip(types, names)
  push!(mystruct_data.fields_names, name)
  push!(mystruct_data.fields_types, type)
end
struct_define(mystruct_data)

mystruct_data = StructConstructor(:Foo, 3)
names = [:a, :b, :c]
types = [:Int64, missing, :String]
for ifield = 1:3
  mystruct_data.fields_names[ifield] = names[ifield]
  mystruct_data.fields_types[ifield] = types[ifield]
end
struct_define(mystruct_data) 

We can more feature such as

• the default value for the fields
• functions that check the values during the definition of an instance or updating

Related packages:

• https://github.com/aminya/AcuteML.jl
• https://github.com/mauro3/Parameters.jl
• https://github.com/cstjean/QuickTypes.jl

Add constructfrom! methods for certain InterfaceTypes

something along the lines of the following would be nice

StructTypes.constructfrom!(::StructTypes.DictType, target::AbstractDict, source) = merge!(target, source)
StructTypes.constructfrom!(::StructTypes.ArrayType, target, source) = append!(target, source)

Define `Struct()` as the struct type for `<:Exception`?

i.e.

StructTypes.StructType(::Type{<:Exception}) = StructTypes.Struct()

I think that's usually the case? (I'm trying to log exceptions as JSON). (This could be superceded by #61, but if we don't go for that, it might still make sense for exceptions).

Combining abstract types and parametric types?

Let's say I have a type hierarchy like this:

abstract type Parent end
mutable struct Child1 <: Parent
    type # Set to 'child1'
end
mutable struct Child2{N<:Integer} <: Parent
    type # Set to 'child2'
    i::N
end

StructTypes.StructType(::Type{Parent}) = StructTypes.AbstractType()
StructTypes.StructType(::Type{Child1}) = StructTypes.Mutable()
StructTypes.StructType(::Type{<:Child2}) = StructTypes.Mutable()

StructTypes.subtypekey(::Type{Parent}) = :type
StructTypes.subtypes(::Type{Parent}) = (child1=Child1, child2=Child2)

Currently, afaik, there is no mechanism for StructTypes to serialize/deserialize the value of N, like how it can write the type key for each child.

Current workarounds I can think of:

• Pick a unique type name for each concrete version of Child2{N} you expect to be possible: (child1=Child1, child2U8=Child2{UInt8}, child2I32=Child2{Int32}, ...)
• Use UnorderedStruct or OrderedStruct instead of Mutable, and let the type parameter be inferred by the constructor.

Would it be feasible to support deserializing type parameters in the same way as the type itself? For example:

struct Child2{N<:Integer} <: Parent
    type # Set to 'child2'
    int_type::Type # Set to string(N)
    i::N
end
StructTypes.parametrickey(::Type{<:Child2}, ::Val{1}) = :int_type

Macros for setting `StructType`?

I find myself typing StructTypes a lot, and was wondering if a macro could be used, for example

struct MyStruct
    val::Int
end
@Struct MyStruct

@Struct struct MyStruct
    val::Int
end

What do people think? I've coded something up in the linked PR for people to play around with.

allow for arbitrary key types by serializing keys aswell

Right now I have some project where I build a very small database by using a Dict that maps singleton types to arbitrary types.

for example:

abstract type BaseType end
struct Sub1 <: BaseType end
struct Sub2 <: BaseType end

tmp = Dict(Sub1()=>"5", Sub2()=>10)

what I'm looking for right now is to have a proper way of serializing and deserializing that. Hooking into the abstract type machinery for BaseType didn't help at all. So I wonder, why don't we have the full StructTypes machinery enabled for all types that currently aren't supported as keys? To me that would be a perfectly valid generalization of the current approach given that the currently supported types, Symbol, String and Int64 are mapped per identity rn afaik.

Error after v1.7 update

I'm getting the following error after updating to StructTypes.jl 1.7.

I'll try to isolate the error with a code example.

LoadError: LoadError: MethodError: construct(::Type{Nothing}, ::Nothing) is ambiguous. Candidates:                                       
  construct(T, ::Nothing; kw...) in StructTypes at /rhome/FNORO/.julia/packages/StructTypes/frYBf/src/StructTypes.jl:504                        
  construct(::Type{T}, x::T; kw...) where T in StructTypes at /rhome/FNORO/.julia/packages/StructTypes/frYBf/src/StructTypes.jl:311             
Possible fix, define                                                                                                                            
  construct(::Type{Nothing}, ::Nothing)                                                                                                         
Stacktrace:                                                                                                                                     
  [1] read(::StructTypes.NullType, buf::Base.CodeUnits{UInt8, String}, pos::Int64, len::Int64, b::UInt8, ::Type{Nothing}; kw::Base.Iterators.Pai
rs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})                                                                                         
    @ JSON3 ~/.julia/packages/JSON3/czMzK/src/structs.jl:154                                                                                    
  [2] read(::StructTypes.NullType, buf::Base.CodeUnits{UInt8, String}, pos::Int64, len::Int64, b::UInt8, ::Type{Nothing})                       
    @ JSON3 ~/.julia/packages/JSON3/czMzK/src/structs.jl:149                                                                                    
  [3] read(::StructTypes.UnorderedStruct, buf::Base.CodeUnits{UInt8, String}, pos::Int64, len::Int64, b::UInt8, ::Type{Any}; allow_inf::Bool, kw
::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})                                                                     
    @ JSON3 ~/.julia/packages/JSON3/czMzK/src/structs.jl:78                                                                                     
  [4] read                                                                                                                                      
    @ ~/.julia/packages/JSON3/czMzK/src/structs.jl:71 [inlined]                                                                                 
  [5] pretty(out::IOBuffer, str::String, indent::Int64, offset::Int64; kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{
}}})                                                                                                                                            
    @ JSON3 ~/.julia/packages/JSON3/czMzK/src/pretty.jl:42

Inverse of `fieldprefix` function

There is fieldprefix function, which transform flat structure into nested objects. Is it possible in some way to have an opposite operation?

As an example, consider following article structure written in json:
Input:

json_str = """
{
  "author": {
      "name": "XXX"
      "link": "https://example.com"
  },
  "title": "YYY",
  "body": "ZZZ"
}
"""

What I want is to put it in flat structure (because in my particular case I do not need author as a separate object)

struct Article
   author_name::String
   author_link::String
   title::String
   body::String
end

And to be able to read articles with the construct like

JSON3.read(json_str, Article)

Relation to `ConstructionBase.jl`

It seems like this package has some overlap with https://github.com/JuliaObjects/ConstructionBase.jl . I'm opening this issue for the sake of cross-referencing.

Make construct available for Date

Thank you for this great package.

I have found a functionality is not support. In my context, I already have Date type column in my data.
I want to construct a field of Date type, but it gives me this expection.

ERROR: MethodError: no method matching construct(::Type{Dates.Date}, ::Dates.Date)
Closest candidates are:
  construct(::Type{T}, ::String; dateformat, kw...) where T<:Dates.TimeType at /home/yuehhua/.julia/packages/StructTypes/LCfaf/src/StructTypes.jl:379
  construct(::Type{T}, ::CategoricalArrays.CategoricalValue{T,R} where R<:Integer) where T at /home/yuehhua/.julia/packages/CategoricalArrays/ZjBSI/src/value.jl:177
  construct(::Any, ::Dict{K,V}; kw...) where {K, V} at /home/yuehhua/.julia/packages/StructTypes/LCfaf/src/StructTypes.jl:279
  ...

Is there any way to fix it?

Use different constructors

Hi

Thanks for this package -- I really enjoy it in conjunction with JSON3.

Consider a simple example of setting up a struct and a mapping.

using JSON3
using StructTypes

struct Foo
    a::Int64
    b::Int64
end

StructTypes.StructType(::Type{Foo}) = StructTypes.Struct()

This provides a way to map from JSON into Foo:

JSON3.read("{\"a\": 1, \"b\": 2}", Foo)

I would like a constructor that sets some fields based on others, like

function Foo(a)
    Foo(a, 2a)
end

Now Foo(1) returns Foo(1, 2) as expected. But I cannot figure out how to map JSON with only an "a". That is, this fails:

JSON3.read("{\"a\": 1}", Foo)

how to serialize subtypes of an abstract type such that deserialization works?

abstract type AbstractT end
struct ConcreteT <: AbstractT end
StructTypes.StructType(::Type{AbstractT}) = StructTypes.AbstractType()
StructTypes.StructType(::Type{ConcreteT}) = StructTypes.Struct()
StructTypes.subtypes(::Type{AbstractT}) = (conc=ConcreteT,)
StructTypes.subtypekey(::Type{AbstractT}) = :subt
JSON3.write(ConcreteT()) #-> "{}"

is `StructTypes.isempty` part of the public API?

I'd like to pirate

StructTypes.isempty(::Missing) = true

but it would be good to know if that's an internal method (and so I should pin the version instead of relying on semver). I suspect the answer is yes by default since it's not documented or exported, so I guess this is also a request that it would be nice to be able to avoid writing fields (with JSON3) which are missing if they show up in StructTypes.omitempties.

Allow use of a function/complex object to pick the `subtype` of an `AbstractType`?

Suppose I have a situation like this:

abstract type MyAbs end

struct A <: MyAbs
    id_field::Tuple{String, Int}
    field_a
end

struct B <: MyAbs
    id_field::Tuple{String, Int}
    field_b
end

And I want to dispatch constructing using id_field[1]. I think currently this isn't supported by the NamedTuple approach of StructTypes.subtypes and StructTypes.subtypekey, but would it make sense to extend them to support such cases?

Should `AbstractUnitRange` be Array-like?

I mean, of course AbstractUnitRange is an array. But unit ranges are not best serialized as arrays: They are usually much more compactly represented as structs. This is especially notable if the range is large.

Example:

julia> JSON3.write(1:20)
"[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]"

I don't know enough about JSON or this package to really understand what the implications of making unit ranges struct-like are, but it sure seems like more efficient serialization to me if only the endpoints could be serialized.

ArrayType serialization with 2D+ arrays

Currently ArrayType() iterates element-wise over an array type using the default iterate(x::T) method when serializing. This has the effect of flattening higher dimensional arrays into vectors. Matrices, for instance, become vectors with no information about their original dimensions.

The JSON standard maintains array order, so it seems safe for me to serialize matrices as vectors of vectors. However, my only way to affect serialization is by overloading the iterate(x::T) function and doing so for the Matrix type seems likely to recompile large parts of Julia base (as well as break my own code.)

Then on deserialization, I would just construct(::Type{Matrix}, x::Vector) = hcat(x...)

It makes sense on some level that this would be natively unsupported because it makes the serialized representation of matrices and vectors of vectors indistinguishable (and thus deserialize(serialize(matrix)) != matrix without an additional construct definition), but that code doesn't work right now either and it seems like there should be some way to for users to enable support for it because matrices are so common. Perhaps there should be something analogous to construct() for serialization?

`omitempties` returning one-element tuple fails

StructTypes.omitempties(::Type{Element}) = (:id)

results in

ERROR: MethodError: no method matching symbolin(::Symbol, ::Symbol)
Closest candidates are:
  symbolin(::Tuple{Vararg{Symbol,N} where N}, ::Symbol) at /Users/user/.julia/packages/StructTypes/CpLmq/src/StructTypes.jl:534
Stacktrace:

zero or 2+ element tuples work fine.

Make NamedTupleType a fully "woven in" StructType?

          Distinct Sorts of StructTypes (today)

description	StructTypes.Types
collective:	DictType, ArrayType
elementary:	StringType, NumberType, BoolType
nondescript	NullType

Dicts accommodate multi-typed keys placed in one-to-one_or_more correspondence with multi-typed values. Often, Dicts are constructed with keys of a single, shared type. It is commonplace to use Dicts where the values share an abstract type or are of a single shared type.
In general, Dicts are not ordered, although an ordering may be imposed if the keys are ordered.

Arrays usually keep some given type of information in each "cell", at each indexable position. Arrays are ordered by indexed enumeration. The content may be orderable through the content type's ordering.

NamedTuples are neither as Dicts nor as Arrays. Their internal content is structurally ordered -- by ordinal position of each field['s name]. External ordering is not readily available for most NamedTuple schema. An external order may be defined insofar as the field's types are individually ordered (or orderable by fiat).

NamedTuples share some collective and some elementary aspects. They are collective "in the small" (within an instance) and elementary "in the large" (over many realizations of one schema). One way
of processing may emphasize functionalities of a collective focus; another computational approach may emphasize elementhood and functional strengths that accompany each NamedTuple instance.

For me to best support StructTypes and other JuliaData packages that rely on StructTypes, NamedTupleType would be included as one of StructType's exported Interface Types.

missing `constructfrom` method for `CustomStruct`?

Perhaps I'm misunderstanding the intention of CustomStruct but I believe it is missing a constructfrom method. Even after defining the required methods I get, e.g.

◖◗ ST.constructfrom(Dict{String,Any}, c)
ERROR: MethodError: no method matching constructfrom(::StructTypes.DictType, ::Type{Dict{String, Any}}, ::Type{String}, ::Type{Any}, ::StructTypes.CustomStruct, ::Column{Int64, Count})
Closest candidates are:
  constructfrom(::StructTypes.DictType, ::Type{T}, ::Type{K}, ::Type{V}, ::StructTypes.DictType, ::Any) where {T, K, V} at ~/.julia/packages/StructTypes/Cmlkm/src/StructTypes.jl:961
  constructfrom(::StructTypes.DictType, ::Type{T}, ::Type{K}, ::Type{V}, ::Union{StructTypes.Mutable, StructTypes.Struct}, ::Any) where {T, K, V} at ~/.julia/packages/StructTypes/Cmlkm/src/StructTypes.jl:978
  constructfrom(::StructTypes.DictType, ::Type{T}, ::Type{K}, ::Type{V}, ::S) where {T, K, V, S} at ~/.julia/packages/StructTypes/Cmlkm/src/StructTypes.jl:958
  ...
Stacktrace:
 [1] constructfrom
   @ ~/.julia/packages/StructTypes/Cmlkm/src/StructTypes.jl:958 [inlined]
 [2] constructfrom(#unused#::StructTypes.DictType, #unused#::Type{Dict{String, Any}}, obj::Column{Int64, Count})
   @ StructTypes ~/.julia/packages/StructTypes/Cmlkm/src/StructTypes.jl:955
 [3] constructfrom(#unused#::Type{Dict{String, Any}}, obj::Column{Int64, Count})
   @ StructTypes ~/.julia/packages/StructTypes/Cmlkm/src/StructTypes.jl:905
 [4] top-level scope
   @ REPL[13]:1

I had defined StructTypes.lowertype(::Type{<:Column}) = Dict{String,Any}, so I was expecting this to work.

Docs are not deploying