Comments (43)
Nitpicking, but how do you expect this example to know to create a new Address from the data under 'addresses'? You can leave the types off of every other field and it would be fine, but List can't work here, at least not with "simple JSON".
from reflectable.dart.
Hi Alan,
In this particular case, I would expect my deserializer to see that Person has a List of addresses. The deserializer should create a new list, and then create new Addresses using the default constructor of Address.
I'm comfortable adding an annotation (as is our customer that I'm chatting with on G+). For example, this would be fine:
@serializable
class Person {
int age;
String name;
List<Address> addresses;
}
from reflectable.dart.
My point is, how does it know to create an Address? You have not told it anywhere that the list "addresses" can be expected to contain addresses.
from reflectable.dart.
Through magic! :)
Ah... I see. OK, we'll have to do List<Address> addresses
. Good catch, thank you!
from reflectable.dart.
Hey Seth, how would you expect to call the serializer/deserializer? Maybe something like this?
// this is a sketch of hypothetical apis using serialization pkg and reflectable
import "package:serialization/json.dart"
show serializable, JsonDeserializer;
@serializable
class Person {
int age;
String name;
List<Address> addresses;
}
@serializable
class Address {
String street;
String zip;
}
main() {
// The generic lets us have the return type for "from", and also lets us
// pass the expected type List<Person>.
var people = new JsonDeserializer<List<Person>>().from('''
[
{"age":21, "name":"Bob", "addresses":[{"street":"123 main", "zip":"88443"}, {"street":"555 second", "zip":"99999"}]},
{"age":3, "name":"Alice", "addresses":[{"street":"222 cedar", "zip":"23456"}]}
]
''');
}
For illustration purposes, something like this would be in the serialization package:
class JsonDeserializer<T> {
T from(String json) {
var obj = JSON.decode(json);
ClassMirror mirror = serializable.reflectType(T);
// <get default constructor, make instance, set fields ... recursively>
return result;
}
}
class Serializable extends Reflectable {
const Serializable() : super(/* ..constrain to capabilities we use ... */);
}
const serializable = const Serializable();
from reflectable.dart.
Sorry that was a wall of code. The main thing I wanted to call out was new JsonDeserializer<List<Person>>()
as a way of providing the expected type. As Alan notes, this is very crucial to do in some way.
from reflectable.dart.
(btw, if you don't mind, I added "dart" after the triple backticks in your code examples, to get pretty colors 🌈 )
from reflectable.dart.
We'd be happy to have to do:
JsonDeserializer<List<Person>>()
or even something like:
JsonDeserializer(create: () => new List<Person>())
I'm flexible here.
w.r.t. ClassMirror mirror = serializable.reflectType(T);
... I assume this is via reflectable and thus will not require MirrorsUsed or bloat the code?
from reflectable.dart.
(Perhaps this is the wrong forum for a long discussion on serialization. but hopefully reflectable can enable simple JSON serialization. It's a good use case.)
from reflectable.dart.
or even something like:
JsonDeserializer(create: () => new List<Person>())
the issue with that is you'd need ctors for each thing, so it would be more like:
JsonDeserializer({ Person: () => new Person(), Address: () => new Address() })
(that's a Map<Type, ConstructorFunction>
) ... starts to feel boilerplate-y.
w.r.t. ClassMirror mirror = serializable.reflectType(T); ... can we do this without code bloat or MirrorsUsed?
Should be able to. The idea behind reflectable
is that you get mirrors by specifying their capabilities, so the mirror only brings in what it needs. What this does mean is, in my sketch above, every @serializable
type would potentially be retained: its default constructor and all of its fields.
It would be awesome if the implementation could follow the reflectType(T)
and based on the generic instantiation, conclude that only List, Person, and Address are constructed -- not other serializable types. Not sure how tricky that would be. But still, even if all serializable
types are retained (default ctor+fields), that would still be a big help IMO.
(Perhaps this is the wrong forum for a long discussion on serialization. but hopefully reflectable can enable simple JSON serialization. It's a good use case.)
Agreed!
from reflectable.dart.
For:
JsonDeserializer({ Person: () => new Person(), Address: () => new Address() })
Yeah, that's not great. The system should auto-detect the default constructor for a type.
from reflectable.dart.
For research, here's some references about serialization in Dart:
- http://stackoverflow.com/questions/27135876/state-of-serialization-and-deserialization-of-json-in-dart
- https://www.dartlang.org/articles/serialization/
from reflectable.dart.
I'm sure these things could be done using Reflectable, because that should be very similar to an approach based on dart:mirrors. However, it may be more straightforward to use a dedicated transformer, because the kind of code that we could generate in a Serializable transformer (or maybe that's already done?) would be so simple:
import 'dart:collection';
import 'dart:convert';
@serializable
class Person {
int age;
String name;
List<Address> addresses;
String toString() => "$name, age $age, living at $addresses";
}
@serializable
class Address {
String street;
String zip;
String toString() => "$street, $zip";
}
// Assumptions: no key is used for each top-level object (map), the
// developer knows what the top-level maps are. The structure of the
// maps is expected to match the structure of the class.
final data = '''
[
{"age":21, "name":"Bob", "addresses":[{"street":"123 main", "zip":"88443"}, {"street":"555 second", "zip":"99999"}]},
{"age":3, "name":"Alice", "addresses":[{"street":"222 cedar", "zip":"23456"}]}
]
''';
// Generated by the Serializable transformer
Address mapToAddress(HashMap map) {
Address address = new Address();
address.street = map['street'];
address.zip = map['zip'];
return address;
}
// Generated by the Serializable transformer
Person mapToPerson(HashMap map) {
Person person = new Person();
person.age = map['age'];
person.name = map['name'];
person.addresses = map['addresses'].map(mapToAddress).toList();
return person;
}
void main() {
JsonDecoder decoder = new JsonDecoder();
List objects = decoder.convert(data);
Iterable<Person> persons = objects.map(mapToPerson);
print(persons);
}
The question is whether there would be a need for more flexible approaches (deserializing into instances of classes that are not statically known to be deserialization targets), but that wouldn't be easily covered by Reflectable, because Reflectable reflection requires metadata (@myReflectable) on the target class.
I'll take a look at the references that came up 3 minutes ago now. ;)
from reflectable.dart.
A transformer is unfortunately not a sufficient solution.
- Transformers require a use of pub build or pub serve.
- Transformers force generated code to be stored in pub serve's memory, which is a deal-breaker if anyone wants to debug their app.
- The API for transformers is not ideal for code generation.
Our users want a simple JSON solution that doesn't require: excessive annotations (a single annotation is sufficient), explicit and manual code generation, code bloat. It should "just work".
I'm very hopefully the reflectable package can provide a solution here. :)
from reflectable.dart.
Reflectable uses a transformer, so if that is a show-stopper then Reflectable cannot be used. With a transformer it should be possible to read/run/debug the transformed code by doing pub build --mode=debug some-directory
and then looking into build/some-directory/..
where the transformed files are stored. (But nobody would ever want to debug their app, of course ;-). True, the API for transformers is not convenient for code generation, but it can be done and it would not affect the users.
Still, Reflectable might be an alternative to dart:mirrors in existing approaches to serializability. If there are some good solutions that just fail to be applicable because they rely on dart:mirrors, it might be helpful to use Reflectable rather than dart:mirrors.
from reflectable.dart.
@eernstg I know this isn't your problem to solve, but the pub build --mode=debug some-directory
workflow isn't sufficient. It doesn't work well with our IDEs and editors, because they are looking at the source directory. It puts all the generated code into a completely separate directory, and creates a parallel source world.
Is the transformer required even with Dartium?
cc @kevmoo who has a source_gen project that might be a better fit.
from reflectable.dart.
I thought the point was that Reflectable is a transformer right now, when it can be experimented with, but would be ultimately incorporated into dart2js code generation. In Dartium it can just fall back to using mirrors.
from reflectable.dart.
I thought the point was that Reflectable is a transformer right now, when it can be experimented with, but would be ultimately incorporated into dart2js code generation. In Dartium it can just fall back to using mirrors.
yeah, I thought so too
from reflectable.dart.
Of course, you never need to consider the output of a translation process if the source language has a well-understood semantics and the translation has a high quality (and tools can consistently support debugging etc. in terms of the pre-translation program). But it is a non-trivial requirement on the tools that they must make Reflectable translation transparent (such that the programmer can use tools showing the pre-translation source program and never even consider the post-translation code). We won't get immediately..
from reflectable.dart.
If it is acceptable in a given context to consider mirrors produced by dart:mirrors APIs as black boxes, then the static mirrors generated by Reflectable might be treated in a similar way, and in that case the programmer would get to see the interfaces (package:reflectable/mirrors.dart, near-identical to the interface of dart:mirrors), and the actual implementation classes would be equally black-boxy with dart:mirrors and with Reflectable. Is that what you are thinking about?
from reflectable.dart.
Yes. My mental model of Reflectable is that it provides a way to specify
what aspects of reflection you want, but that then your usage is the same
as if you were using mirrors. That includes being black-box in that they
don't require the use of pub serve or build.
On Tue, Mar 10, 2015 at 10:27 AM Erik Ernst [email protected]
wrote:
If it is acceptable in a given context to consider mirrors produced by
dart:mirrors APIs as black boxes, then the static mirrors generated by
Reflectable might be treated in a similar way, and in that case the
programmer would get to see the interfaces (package:reflectable/mirrors.dart,
near-identical to the interface of dart:mirrors), and the actual
implementation classes would be equally black-boxy with dart:mirrors and
with Reflectable. Is that what you are thinking about?—
Reply to this email directly or view it on GitHub
#4 (comment).
from reflectable.dart.
About "[the Reflectable transformer] puts all the generated code into a completely separate directory": If we want to have transformer-specific tool support, then it would probably not be impossible to offer the choice in IDEs etc.: "Debug the transformed program" versus the normal "Debug the program". With Reflectable, the pre-transform program should have exactly the same semantics (apart from the different implementation of operations in dart:mirrors based pre-transform reflection and static mirror based post-transform reflection), so with a well maintained Reflectable library it should be a matter of working in the pre-transform world and then checking that it works the same in the transformed world.
from reflectable.dart.
About "usage is the same": Nearly. It differs in little things like 'myReflectable.reflect(o)' rather than 'reflect(o)'. But it should be easy to port.
from reflectable.dart.
the pre-transform program should have exactly the same semantics
This is a great goal, but there will always be weird cases where semantics don't match, especially when compiling to JavaScript. Especially when building for the web and all the various different browser nuances, being able to inspect the generated code and step through it is critical.
from reflectable.dart.
Seth: Does that mean you're saying you require a solution that generates
Dart code ahead of time? And, presumably, that you find mirrors
unacceptable in principle, even if they worked fine with dart2js tree
shaking? It seems like Reflectable, in any implementation, wouldn't fit
with your requirements if that's true.
On Tue, Mar 10, 2015 at 10:57 AM Seth Ladd [email protected] wrote:
the pre-transform program should have exactly the same semantics
This is a great goal, but there will always be weird cases where semantics
don't match, especially when compiling to JavaScript. Especially when
building for the web and all the various different browser nuances, being
able to inspect the generated code and step through it is critical.—
Reply to this email directly or view it on GitHub
#4 (comment).
from reflectable.dart.
If it works with Mirrors for Dartium/VM, that's fine, as long as any code bloat is removed before we compile to JavaScript.
We'd also have to ensure that source maps work excellently well, when a developer is debugging their app in various browsers. The source map should point the user back to code they can see and inspect in their editors.
from reflectable.dart.
With regards to constructors, it'd be nice if no constructor is called at all. The idea behind serialization (in most cases) is to store state of an object and restore the exact same state at a later time. Since constructors can have logic, executing them could alter the state of the restored object. I know that GSON (Java) uses Unsafe.allocateInstance to accomplish this. I'm curious if the same thing could be accomplished in Dart. I assume the problem would be with accomplishing this in JavaScript via dart2js.
For what it's worth, a first party mirrors based approach for Dartium and the VM sounds awesome. I'd also prefer to see the conversion to JavaScript happen in dart2js instead of a transformer. Having to rely on pub serve can be annoying if your app is set up to serve its own content and you want to do development for it when it's running that way. This scenario forces you to have to stop, build, and run again manually. Note that pub serve is wonderful when doing client only development.
from reflectable.dart.
Regardless of whether it's a good idea or not, Dart does not have a way of
allocating an object without calling a constructor.
On Tue, Mar 10, 2015 at 4:01 PM Jim Simon [email protected] wrote:
With regards to constructors, it'd be nice if no constructor is called at
all. The idea behind serialization (in most cases) is to store state of an
object and restore the exact same state at a later time. Since constructors
can have logic, executing them could alter the state of the restored
object. I know that GSON (Java) uses Unsafe.allocateInstance to accomplish
this. I'm curious if the same thing could be accomplished in Dart. I assume
the problem would be with accomplishing this in JavaScript via dart2js.For what it's worth, a first party mirrors based approach for Dartium and
the VM sounds awesome. I'd also prefer to see the conversion to JavaScript
happen in dart2js instead of a transformer. Having to rely on pub serve can
be annoying if your app is set up to serve its own content and you want to
do development for it when it's running that way. This scenario forces you
to have to stop, build, and run again manually. Note that pub serve is
wonderful when doing client only development.—
Reply to this email directly or view it on GitHub
#4 (comment).
from reflectable.dart.
Any approach that involves generated code could generate a constructor for each @serializable class that takes a representation of an object state and creates an object in that state.
from reflectable.dart.
I think my last comment got a little carried away. The important part of what I was trying to say was that it'd be ideal to avoid side effects when restoring state.
Generating a constructor is fine for output from dart2js, but I'd personally prefer the default constructor and mirrors approach when it comes to Dartium and the VM. I think the default constructor approach is common enough in other libraries (at least in java) that as long as it's documented it won't be a problem.
Also I'm assuming this won't be able to handle properties with type "dynamic" since the type at runtime could be anything, right?
from reflectable.dart.
@jimsimon What makes you think that default constructors will play nice? ;-) For instance, objects with final
fields will require them to be initialized during construction, and the default constructor may not choose the values you need (and dart:mirrors will say "..has no instance setter.." if you try to correct them post-hoc).
from reflectable.dart.
In Dart not only can the default constructor have side effects, so can
setting fields. And doing it through mirrors does not give you away around
that, unless you go through some gymnastics to get at the private
underlying fields.
But this is talking about extremely restricted objects, and the restriction
that they can be effectively recreated by calling a default constructor and
then setters on fields whose names correspond to the map entries is a
reasonable assumption, and less than some. And yes, they can't handle
entries typed dynamic, or whose generics are dynamic, or whose types are
not completely concrete (i.e. where the values might be things that
implement the interface or are subclasses), the object graph is acyclic,
and probably other restrictions as well.
On Wed, Mar 11, 2015 at 6:17 AM Erik Ernst [email protected] wrote:
@jimsimon https://github.com/jimsimon What makes you think that default
constructors will play nice? ;-) For instance, objects with final fields
will require them to be initialized during construction, and the default
constructor may not choose the values you need (and dart:mirrors will say
"..has no instance setter.." if you try to correct them post-hoc).—
Reply to this email directly or view it on GitHub
#4 (comment).
from reflectable.dart.
Is this package far enough along that I could try spiking out a serializer/deserializer this weekend?
Also, it may be possible to handle dynamic types as lists and maps based on whether the property's value starts with a { or a [. It's not a perfect restoration of the original object, but it could be a workable fallback. Either falling back to lists and maps or erroring out and saying "dynamic types aren't supported" work for me.
Other serializers have ways of handling polymorphism, but a 1.0 release of this serialization library wouldn't necessarily have to. Some of these methods might allow dynamic types to work as well. See http://programmerbruce.blogspot.com/2011/05/deserialize-json-with-jackson-into.html for one way to handle this.
from reflectable.dart.
No, unfortunately you cannot test anything like a serializer/deserializer with the current Reflectable package, it is not sufficiently complete for that. But we are working on it all the time. ;)
from reflectable.dart.
Polymophism would be necessary for me, it's easy to do with dart:mirrors
by storing the library and class in the JSON, and then using currentMirrorSystem().findLibrary(librarySymbol).declarations[typeSymbol]
. Would this use case be possible with reflectable?
from reflectable.dart.
Trying to recover the context: I'm not sure what it means to 'need polymorphism' in the context of (de)serialization, but I suppose it might mean "I want to obtain support for (de)serializing objects which are instances of subtypes of a given type T without giving annotations or similar explicit directions concerning any other types than T" (so (de)serialization for the subtypes should be handled automatically). I would never expect the receiver of a serialized value to know statically which type of object it is receiving on a given socket or similar input channel (that should be specified in the serialization itself), but Dart's compilation model makes it reasonable to require that it must be an instance of a class C
which is a member of a statically known set of classes. That set could be enumerated directly (by having annotations on every member of the set), or it could be obtained by quantification (as in "please support serialization for all subtypes of C
"). So I'll assume that we are talking about the situation where librarySymbol
and typeSymbol
belong to statically known finite sets of symbols; typically we'll just support all libraries in the program, but only some of the types.
With that, you would have to use librarySymbol
and typeSymbol
values belonging to the set that you have support for; other than that, there is nothing stopping you from doing
myReflectable.findLibrary(librarySymbol).declarations[typeSymbol]
to get a ClassMirror
for the class named typeSymbol
in the library named librarySymbol
. We haven't yet implemented anything in the area of class mirrors, but I don't foresee any particular difficulties in supporting them just like dart:mirrors
, that's actually just a lot of static information delivered using statically generated maps etc.
from reflectable.dart.
Yes I understand that the libraries and classes available must be annotated. The use case I'm thinking about is if there are multiple subclasses of Address and addresses contains a mix of those.
@serializable
class Address {
String street;
String zip;
String country;
String toString() => "$street, $zip, $county";
}
@serializable
class StateAddress extends Address {
String state;
String toString() => "$street, $zip, $state $county";
}
final data = '''
[
{"age":21, "name":"Bob", "addresses":[{"street":"123 main", "zip":"88443", "country": "SE", "@type": "Address"}, {"street":"555 second", "zip":"99999", "country": "US", "state": "OH", "@type": "StateAddress"}]}
]
''';
After I have deserialized that JSON I would expect a list with one instance of Address and one instance of StateAddress so that toString()
prints the correct format.
from reflectable.dart.
That's exactly what I meant: 'that [i.e., the type of the incoming object] should be specified in the serialization itself' is satisfied in your example by the bindings on the form "@type" : "SomeClass"
, so you would certainly get an instance of Address
and an instance of StateAddress
from the deserializer, as long as the serialization package you are using will provide and interpret these bindings.
There is no "@type": "Person"
binding in the top-level element in data
; if you want to discover that this is a Person
without adding such a binding we're in a different setting, but I assume that it was omitted by accident.
from reflectable.dart.
If you want to discover that this is a Person without adding such a binding we're in a different setting, but I assume that it was omitted by accident.
The point was the Address classes so I ignored it, might have been dumb tough as it causes more confusion.
from reflectable.dart.
OK! Generating and interpreting those "@type": "SomeClass"
bindings is handled by some serialization library L in any case, so it's only relevant to the design of Reflectable to the extent that L needs something very special from Reflectable in order to do that (which does not seem likely).
from reflectable.dart.
I have added a small serialization example, showing what we hope to achieve (https://github.com/dart-lang/reflectable/blob/master/test_reflectable/test/serialize_test.dart, https://github.com/dart-lang/reflectable/blob/master/test_reflectable/lib/serialize.dart).
It already runs in the untransformed case (relying on dart:mirrors) and I hope to have added enough functionality to the transformer very soon (https://codereview.chromium.org/1181153003, https://codereview.chromium.org/1182083002, https://codereview.chromium.org/1181413005) that it can also run transformed.
However I am running into trouble with the way transformers only work on one package at a time (dart-lang/sdk#17306). So it is currently not possible to write have the Serializable extends Reflectable
defined in one package, and then use it in another. Hopefully this will improve when dart-lang/sdk#17306 is resolved.
from reflectable.dart.
Thanks for all your input! We'll close this issue now. In case the test serialize_test.dart
does not adequately embody the requirements coming from this application area it should be raised as a new issue.
from reflectable.dart.
LGTM!
(For those interested, here's the link for test_reflectable/lib/serialize.dart and
test_reflectable/test/serialize_test.dart)
from reflectable.dart.
Related Issues (20)
- [question] - Is it possible to have Annotations with Parameters? HOT 2
- [Question] - how to get Constructor arguments? HOT 2
- [Bug?] ERROR: This requires the 'constructor-tearoffs' language feature to be enabled. HOT 12
- [Question] - Listening to Method invocation HOT 6
- [Question] - Stats on Bundle Sizes and Performance Implications/metrics of using Reflectable ? HOT 1
- Cannot reflect ```flutter/material.dart``` URIs for ```Container``` type HOT 12
- This reflector does not match anything HOT 11
- Reflectable is not a know builder HOT 1
- Unsupported operation: location HOT 4
- Reflectable don't find "main" but im using in my library not an application HOT 5
- How to generate code according to filter rules HOT 2
- The build_runner those not generate anything HOT 1
- Annotated parameter and Functions
- Include annotated types that are not imported to `main.dart` HOT 2
- Severe error during build with @reflector and @JsonKey on the same class. HOT 1
- isSubtype exception for classes without parents HOT 2
- Where is initializeReflectable method? HOT 3
- Error: Reflecting on un-marked type "Type" HOT 2
- bug: issue with using mixins HOT 5
- importing private libaries during build_runner build HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from reflectable.dart.