Comments (5)
Thanks for opening this. Here's a complete reproducer:
import msgspec
from typing import Optional
class Wrapper(msgspec.Struct):
result: Optional[msgspec.Raw]
msg = b'{"result": {"foo": "bar"}}'
print(msgspec.json.decode(msg, type=Wrapper))
#> Traceback (most recent call last):
#> File "/home/jcristharif/Code/msgspec/bug.py", line 11, in <module>
#> print(msgspec.json.decode(msg, type=Wrapper))
#> msgspec.ValidationError: Expected `null`, got `object` - at `$.result`
from msgspec.
Can you say more about your use case for the Optional[Raw]
annotation? How do you plan on consuming this field post decode (or producing this field for encoding)?
The current Raw
decoding logic assumed that Raw
nodes won't ever be included in a union like Raw | None
(same as Optional[Raw]
) . What you're seeing here is a bug in this limitation actually being enforced.
That said, there's a few things we could do here:
- Error if
Raw
is ever present in aUnion
. The type described above would error saying you can't includeRaw
inside a union likeOptional[Raw]
. - Support
Raw
inside unions, but ifRaw
is ever present then we always decode it as aRaw
value, even if it matches other types in the union. There's some precedence for this - this is how we handleAny
today. In your case abovemypy
would let you pass aNone
orRaw
toresult
, but when decoding you'd always end up with aRaw
, even if the JSON value isnull
(in that case you'd getRaw(b'null')
. The union support would then be most useful for cases where you might want to support passing in-memory objects when programatically constructing these, but on the decode-side always decode them asRaw
. Raw
is not supported in unions except for withNone
. Soresult: Raw
orresult: Optional[Raw]
both work, butresult: Raw | int
would error as an unsupported type. This restriction matches what we already do for custom types. In this casenull
would decode asNone
, but any other value would decode as aRaw
value.- A mix of 2 and 3. In this case any union with
Raw
would be supported, but onlyRaw
andNone
would actually be decoded, everything else would silently be ignored (like in 2).
Given these options, I'm leaning towards either 1 (easiest, best matches the main use case for Raw
), or 3 (slightly harder, but may still be useful and matches a convention we had for other types). 2 also seems fine. I think option 4 would be confusing.
from msgspec.
from msgspec.
If I get a vote, it's for "Raw | None"
Sorry, 2, 3, or 4 could support that type, the question is what happens on decode.
# Given the following type definition
import msgspec
class Wrapper(msgspec.Struct):
result: msgspec.Raw | None
# And the message
msg = b'{"result": null}'
# The decode call
msgspec.json.decode(msg, type=Wrapper)
# would result in the following values:
# 1. Would error as an unsupported type
# 2. Would decode as `Wrapper(result=Raw(b'null'))`
# 3. Would decode as `Wrapper(result=None)`
# 4. Would decode as `Wrapper(result=None)`
As an aside, your use case would probably be best supported by not using Raw | None
, but rather having result
default to an empty Raw()
instance. The following works today:
import msgspec
from typing import Optional
class Wrapper(msgspec.Struct):
# result is always a Raw type, will use an empty Raw if not present
result: msgspec.Raw = msgspec.Raw()
# you didn't say what the structure of `error` was, so guessing string
error: str = ""
msgs = [
b'{"result": {"x": 1}}', # result present
b'{"result": null}', # result explicitly null
b'{"error": "an error message"}', # result missing, error present
]
for msg in msgs:
obj = msgspec.json.decode(msg, type=Wrapper)
# An empty `msgspec.Result()` is false-y, so you can branch on if
# this field is Truthy to detect the presence of a result.
if obj.result:
print(f"result: {bytes(obj.result).decode()!r}")
else:
print(f"error: {obj.error!r}")
#> result: '{"x": 1}'
#> result: 'null'
#> error: 'an error message'
from msgspec.
from msgspec.
Related Issues (20)
- Docs page on testing
- json schema generation - differences between pydantic and msgspec HOT 3
- Allow conversion to collection from generator HOT 2
- Porting guide for users coming from `orjson`
- Converting dicts into list with key-reuse HOT 3
- Collecting multiple validation/constraint errors at once HOT 1
- Allow `omit_defaults` to exclude fields when encoded value is `{}` (empty dict)
- Duplicate key detection
- Allow unknown tags, defaulting to tagged base
- Implementing optional bytes type for json. HOT 1
- Update annotation parsing to work with PEP 649 in Python 3.13
- `omit_defaults` does not omit tuples and frozensets HOT 2
- Field Alias Overrides in Subclasses Not Reflected in __struct_encode_fields__
- Convert builtin types to numpy HOT 2
- Is it possible to have the decoding of union of all subclasses of a struct
- Subclasses of frozen Structs causing mypy error: `Cannot inherit non-frozen dataclass from a frozen one` HOT 2
- Datetime without timezone are decoded as str with msgspec.msgpack. in 0.18.6 HOT 1
- Struct import fails in pytest - Missing requirement argument 'attribute x' or unhashable type: 'StructClass'
- Generate FastAPI docs from msgspec.Struct json schema HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from msgspec.