lwabeke / unmarshal.jl Goto Github PK

Implements a Julia Unmarshal package, which will construct Julia objects from a marshalled string after having been parsed by another package (currently JSON.jl and LaxyJSON.jl has been tested).

License: Other

Julia 100.00%

unmarshal.jl's People

Contributors

Stargazers

Watchers

Forkers

tkelman rwfocke non-jedi sunoru dhairyalgandhi

unmarshal.jl's Issues

Issue while unmarshalling a structure with a dictionary

Hi,

When I use the library to unmarshall structures that have dictionaries in them, an error occurs with message:
ArgumentError: Key ndel is missing from the structure Dict{String,String} and field is not Nullable
in unmarshal at Unmarshal/src/Unmarshal.jl:67
in unmarshal at Unmarshal/src/Unmarshal.jl:91
in unmarshal at Unmarshal/src/Unmarshal.jl:86

To reproduce:
mutable struct DictTest
testDict::Dict{String, String}
end
dictTest = DictTest(Dict{String, String}("a" => "Test1", "b" => "Test2"))
Unmarshal.unmarshal(DictTest, JSON.parse(JSON.json(dictTest)))

Great library btw, thanks for writing!

Unmarshal issue with v0.4.1

Hi,

If we try this code with the latest tag v0.4.1, it produces an error:

using Unmarshal
mutable struct TestStruct
    test::Symbol
end
packedDict = Dict{String,Any}("test" => "testValue")
packedVND = Unmarshal.unmarshal(TestStruct, packedDict)

ERROR: ArgumentError: no unmarshal function defined to convert String to Symbol; consider providing a specialization
Stacktrace:
 [1] unmarshal(::Type{T} where T, ::String, ::Bool, ::Int64) at /home/gearsad/.julia/packages/Unmarshal/2LYo9/src/Unmarshal.jl:234
 [2] unmarshal(::Type{T} where T, ::Dict{String,Any}, ::Bool, ::Int64) at /home/gearsad/.julia/packages/Unmarshal/2LYo9/src/Unmarshal.jl:127
 [3] unmarshal(::Type{T} where T, ::Dict{String,Any}) at /home/gearsad/.julia/packages/Unmarshal/2LYo9/src/Unmarshal.jl:99
 [4] top-level scope at REPL[4]:1
 [5] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1088

This works with v0.4.0, so it seems the latest release introduced an issue?

Add support for keyword based constructor and default values (Base.@kwdef)

Hi,
there are cases where not all keys are available in JSON object. Typical situation is using JSON format in configuration files.
Therefore, it would be valuable to add support for keyword based constructor and default values (Base.@kwdef).

I've went through code and at least by my limited Julia knowledge there is not much to do. In fact I prototyped this change and works fine. What is unclear to me, is how to identify if struct constructor is keyword based or no. Quick check (also ask on discord) tell me that there is not an option to check this. Therefore a solution is in a way that an optional parameter, like defaultValues=false is added to Unmarshal function definition. The parameter defines if struct supports keywords. Using this approach a backward compatibility is retained.
@lwabeke what are yout thoughts? Should we go in this direction?

Simon

no unmarshal function defined to convert Nothing to String?

Hi,
I have 2 struct defined as follows One is tweet and another is user of the tweet. The tweet attribute/field place can have some string value or nothing

struct UserA
    id::Union{Int64, Nothing}
    id_str::Union{String, Nothing}
    created_at::Union{String, Nothing}
    name::Union{String, Nothing}
end


struct TweetB
    id::Union{Int64, Nothing}
    id_str::Union{String, Nothing}
    created_at::Union{String, Nothing}
    place::Union{String, Nothing}
    user::UserA
end

Now I am trying to unmarshall the following json:

json_str = "{\"id_str\":\"1305501948074835974\",\"created_at\":\"Mon Sep 14 13:41:34 +0000 2020\",\"place\":null,\"id\":1305501948074835974,\"user\":{\"name\":\"Donald J. Trump\",\"id_str\":\"25073877\",\"created_at\":\"Wed Mar 18 13:46:38 +0000 2009\",\"id\":25073877}}"

parsed_json = JSON.parse(json_str)

a_tweet = Unmarshal.unmarshal(TweetB, parsed_json)

I get the following error:

ERROR: ArgumentError: no unmarshal function defined to convert Nothing to String; consider providing a specialization

As the place field has value nothing after parsing now It can not unmarshal the result. Is that a bug or I am doing some mistake.
Thanks

Error tagging new release

The tag name "0.1.2" is not of the appropriate SemVer form (vX.Y.Z).
cc: @lwabeke

Unmarshalling with Dictionary issue

Hi,

We just upgraded to 0.1.1 and see an issue marshalling dictionaries. This may be similar to #9 ?

Unmarshal is unable to convert the dictionary in this example:

using JSON
using Unmarshal

mutable struct TestUnmarshal
  a::String
  b::String
  links::Dict{String, String}
end

raw = "{\"a\": \"\",\"b\": \"Test\",\"links\": {\"self\": \"TestDict\"}}"
j = JSON.parse(raw)
@show val = Unmarshal.unmarshal(TestUnmarshal, j)

ERROR: Unexpected character
Line: 0
Around: ...self...
            ^

Stacktrace:
 [1] _error(::String, ::JSON.Parser.MemoryParserState) at /home/gearsad/.julia/v0.6/JSON/src/Parser.jl:140
 [2] parse_jsconstant(::JSON.Parser.MemoryParserState) at /home/gearsad/.julia/v0.6/JSON/src/Parser.jl:186
 [3] parse_value(::JSON.Parser.ParserContext{Dict{String,Any},Int64}, ::JSON.Parser.MemoryParserState) at /home/gearsad/.julia/v0.6/JSON/src/Parser.jl:170
 [4] #parse#1(::Type{Dict{String,Any}}, ::Type{Int64}, ::Function, ::String) at /home/gearsad/.julia/v0.6/JSON/src/Parser.jl:393
 [5] unmarshal(::Type{Dict{String,String}}, ::Dict{String,Any}, ::Bool, ::Int64) at /home/gearsad/.julia/v0.6/Unmarshal/src/Unmarshal.jl:143
 [6] unmarshal(::Type{T} where T, ::Dict{String,Any}, ::Bool, ::Int64) at /home/gearsad/.julia/v0.6/Unmarshal/src/Unmarshal.jl:118
 [7] unmarshal(::Type{T} where T, ::Dict{String,Any}) at /home/gearsad/.julia/v0.6/Unmarshal/src/Unmarshal.jl:94
 [8] macro expansion at /home/gearsad/.julia/v0.6/Atom/src/repl.jl:118 [inlined]
 [9] anonymous at ./<missing>:?

Thanks again for building this library, it's saving a lot of work not having to manually convert JSON structs!

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

ERROR: LoadError: UndefVarError: Associative not defined

Happens while testing the package on julia 1.0
This looks fixed on master, so a new release will make Unmarshal 1.0-compatible :)

Review comments from JSON.jl

I hope to look over the code and make a PR with the remaining review comments soon. Will update this thread when I do.

Incorrect handler of Tuple types

The unmarshal function for Tuple types cannot handle Tuple fields or Tuple of values in different types correctly. And the current implementation will unmarshal a tuple of several elements (e.g. in Int type) with a Tuple{Int} (actually, any Tuple types whose first type parameter is Int), which is inconsistent IMO because Tuple{Int} means the type of tuples that contain only one Int. Instead, it should be NTuple{N, Int} where N.
For example:

julia> import Unmarshal, JSON

julia> struct Foo
       x::Tuple
       end

julia> foo = Foo((1,5,6))
Foo((1, 5, 6))

julia> Unmarshal.unmarshal(Foo, JSON.parse(JSON.json(foo)))
ERROR: StackOverflowError:
Stacktrace:
 [1] unmarshal(::Type{Any}, ::Int64, ::Bool, ::Int64) at C:\Users\sunoru\pro\julia-packages\Unmarshal\src\Unmarshal.jl:147 (repeats 80000 times)

julia> t = (1,6,7)
(1, 6, 7)

julia> Unmarshal.unmarshal(Tuple{Int}, JSON.parse(JSON.json(t)))
(1, 6, 7)

julia> Unmarshal.unmarshal(typeof(t), JSON.parse(JSON.json(t)))
(1, 6, 7)

julia> t2 = (1, 3, 4.5)
(1, 3, 4.5)

julia> Unmarshal.unmarshal(typeof(t2), JSON.parse(JSON.json(t2)))
ERROR: InexactError: Int64(4.5)
Stacktrace:
 [1] Type at .\float.jl:703 [inlined]
 [2] unmarshal at C:\Users\sunoru\pro\julia-packages\Unmarshal\src\Unmarshal.jl:144 [inlined]
 [3] (::getfield(Unmarshal, Symbol("##5#6")){Tuple{Int64,Int64,Float64},Bool})(::Float64) at .\none:0
 [4] iterate(::Base.Generator{Array{Any,1},getfield(Unmarshal, Symbol("##5#6")){Tuple{Int64,Int64,Float64},Bool}}, ::Int64) at .\generator.jl:47
 [5] append_any(::Any, ::Vararg{Any,N} where N) at .\essentials.jl:680
 [6] unmarshal(::Type{Tuple{Int64,Int64,Float64}}, ::Array{Any,1}, ::Bool, ::Int64) at C:\Users\sunoru\pro\julia-packages\Unmarshal\src\Unmarshal.jl:117
 [7] unmarshal(::Type{Tuple{Int64,Int64,Float64}}, ::Array{Any,1}) at C:\Users\sunoru\pro\julia-packages\Unmarshal\src\Unmarshal.jl:112
 [8] top-level scope at none:0

I'm going to open a pull request to help fix this. Just want to file an issue first to let you know :)

Unmarshaling empty matrix results in ERROR: MethodError: no method matching

Hi
it looks there is an corner case not supported with unmarshal.
It happens where an empty Matrix is unmarshaled. A JSON matrix value [] is parsed to Any[]. Next, the unmarshaling of the form
Unmarshal.unmarshal(Matrix{Int32}, Any[]) returns Any[]. Note that: typeof(Any[]) returns Vector{Any}. However, constructor for Matrix expects the type Matrix{Any}

MWE demonstrates this:

struct test
    id::Int32
    cs::Array{Int32,2} 
end

julia> d1=test(3, reshape([],0,0))
test(3, Matrix{Int32}(undef, 0, 0))

julia> d2=JSON.json(d1)
"{\"id\":3,\"cs\":[]}"

julia> d3=JSON.parse(d2)
Dict{String, Any} with 2 entries:
  "cs" => Any[]
  "id" => 3

julia> d4=Unmarshal.unmarshal(test,d3,true)
test AbstractDict
\--> id <: Int32
\--> cs <: Matrix{Int32}
        Array{Int32, 2}
ERROR: MethodError: no method matching Matrix{Int32}(::Vector{Any})
Closest candidates are:
  Array{T, N}(::AbstractArray{S, N}) where {T, N, S} at C:\Programs\Julia-1.7.0\share\julia\base\array.jl:563
  Matrix{T}(::LinearAlgebra.UpperTriangular) where T at C:\Programs\Julia-1.7.0\share\julia\stdlib\v1.7\LinearAlgebra\src\triangular.jl:177
  Matrix{T}(::LinearAlgebra.UpperHessenberg) where T at C:\Programs\Julia-1.7.0\share\julia\stdlib\v1.7\LinearAlgebra\src\hessenberg.jl:68
  ...

I'd be glad if this could be fixed.

Simon

Unmarshalling to a Julia type when type has Union Values?

Hi,
I am trying to unmarshal a json to a Struct type where there is Union in type attributes but it does not work. Is there a way to do that?

That function works:

input = "{ \"id\": 25073877, \"id_str\": \"25073877\"}"

struct UserX
    id::Int64
    id_str::String
end

Unmarshal.unmarshal(UserX, JSON.parse(input))

But Having Union in the struct Marshalling is not working:

struct UserY
    id::Union{Int64, Nothing}
    id_str::Union{String, Nothing}
end

Unmarshal.unmarshal(UserY, JSON.parse(input))

this through following error:

ERROR: MethodError: unmarshal(::Type{Union{Nothing, String}}, ::String, ::Bool, ::Int64) is ambiguous. Candidates:
  unmarshal(DT::Type, parsedJson::String, verbose::Bool, verboseLvl::Int64) in Unmarshal at /home/datapsycho/.julia/packages/Unmarshal/G72dP/src/Unmarshal.jl:38
  unmarshal(::Type{Union{Nothing, T}}, x, verbose::Bool, verboseLvl::Int64) where T in Unmarshal at /home/datapsycho/.julia/packages/Unmarshal/G72dP/src/Unmarshal.jl:232
Possible fix, define
  unmarshal(::Type{Union{Nothing, T}}, ::String, ::Bool, ::Int64) where T

Is it possible to unmarshal such data?