Generate Java classes from JSON Schema about json-schema-validator

[fge]

OK, 2.0.0 is out, the core is out and in good shape, and jsonschema2pojo is there as well; so, for the initial demand, I guess a sample processor can be easily written.

By the way, jsonschema2pojo has an example site:

http://jsonschema2pojo.org

As I have a compiler processor already, I guess this means I can write such a processor quite quickly!

@dportabella would you mind sending me a sample schema which you would like a POJO out of? This will help me inject an example on the web site.

[fge]

OK, Cc'ing two people there:

• @cowtowncoder, since he is the author of Jackson, which I heavily depend on;
• @reinert, since his project (https://github.com/JJSchema) does the reverse (generate schemas from annotated POJOs).

If I understand correctly you'd like some equivalent of xmlbeans. But that is territory I have never ventured into. I knew of jsonschema2pojo already, and I don't know the status of this project.

More generally, my objective, which I am coding right now, is to transform the API so that it be possible to create "JSON processing chains": the processing chain for validation is ref resolution, syntax checking, instance validation; I and @reinert have future plans to use this common base so that you could prepend a "class to schema" processor in front of this chain and directly validate JSON inputs from a class. @cowtowncoder maybe has such a chain in reserve, which is why I Cc: him.

And you'd want a chain which does JSON Schema -> POJO. This is a very interesting prospect, but as I said, I have never had an attempt at it. I'll throw an eye at jsonschema2pojo which indeed looks like the closest project to do what you need. If I can integrate this some day, I certainly will!

[cowtowncoder]

I know this is familiar to @fge but I'll mention it so others are also aware: Jackson 2.1 added new extension point for accessing introspected POJO structure via ObjectMapper.acceptJsonFormatVisitor() call.
So one possibility might be to make a way to "traverse" JSON Schema similarly, to produce equivalent callbacks, and perhaps use this as a pipeline. Being callback-based, it might lends itself to pipelining.

[fge]

@cowtowncoder in fact, I know it exists, but I am not familiar with it. Is there a simple usage example? I'd like to avoid reinventing the wheel in my own project...

[cowtowncoder]

Best example is probably the external schema generator:

https://github.com/FasterXML/jackson-module-jsonSchema

but Avro module also uses it for building Avro schemas from POJOs:

https://github.com/FasterXML/jackson-dataformat-avro

Also; worth nothing that 2.2 has stream-lined, cleaned up version, so no new code should be written against 2.1 version of API ... was bit experimental, and with experience a few issues surfaced, fixed in 2.2. :-)

[reinert]

Well, I tryed to figure out how to extend this bases classes but I couldn't see how does it works.
Could you explain us in few words what's the purpose of the design and how can we entend it to implement new features?

For instance, I could not see where do you verify that the property of some Pojo has an required attribute and binds it.

[cowtowncoder]

It's a callback/visitor based interface which traverses a single POJO definition, including properties (and recursively things reached, depending on return values). As such visitors need to collect or use any information they want; information is exposed by handing objects (like BeanPropertyWriter that represents property from serialization point of view).

So this is not used for binding, but for exposing information that data-binding (and other use cases) use. It was designed to support case of JSON Schema generation initially, and has been used for Avro Schema generation. Intent was to make it generic enough to hopefully be useful for other uses as well, but it is expected that more information needs to be exposed, based on specific requests.

I don't know exactly how it would be used for this feature, but it would seem relatively simple to build a Java source code generator, for example, although it would initially seem useless (since Jackson starts with POJO definition).
But the other part would then be some other source to use instead of Jackson core, which would produce callbacks based on another information source. Such as, say, a JSON Schema. So callback interface can be used to connect various sources (of POJO structure information) and sinks (generators of some kind).

[timolson]

I also have a similar use case where I need a visitor to every node in a schema. My old code used Jackson plus manual $ref resolution, but it would be great to have a generic visitor pattern (not just single keyword-based) over the stitched ($ref resolved, syntax-validated) schema.

[fge]

@timolson what do you need to visit a schema for exactly?

[timolson]

We have a couple use cases, but the easiest to understand is example generation. We add an "example" keyword to our schema and use a visitor pattern to generate an example JSON instance from the enhanced schema. We do things like randomly select from the given 'enum', or generate a random integer within the min/max range, or simply copy the data from the "example" field in the schema. So we can't really just use a keyword visitor, because we work with many keywords on each node. It's not really syntax validation – we'd like to execute our visitor after the schema itself has been fully validated. It's not instance validation, either, since we are creating an instance from schema walking, not validating an existing instance.

[fge]

OK, I see...

Well, with the new core I have created, you have the option to somehow visit a schema with your own processor by using the .append() and .setPointer() operations on SchemaTree. This is how, for instance, I validate syntax and keywords such as allOf or anyOf. I could write a small, demo processor as an example and extract, for instance, all example keywords, print where they are found and their contents.

[dportabella]

@fge, that's great!

this one:
#36 (comment)

and it would be great if it produces the POJOs with the Jackson annotations JsonTypeInfo and JsonSubTypes as found in this example:
Example 4: Simple Deserialization/Serialization To/From Container Object With Polymorphic Collection
http://programmerbruce.blogspot.de/2011/05/deserialize-json-with-jackson-into.html

[fge]

@dportabella first the schema needs to be rewritten; definitions will need to be used. And extends does not "fuse" schemas... I first need to write a schema. "additionalProperties": false is going to be the main problem here! Is it really a requirement? Allowing (but ignoring) additional properties would be much more simple. In fact, all IETF specifications describing JSON formats (JSON Schema, JSON Patch, JSON Home, others) do that. Stay tuned while I write a schema -- which will use an invention of mine...

@joelittlejohn, how easy is it to inject support for new annotations in jsonschema2pojo? This will be needed here.

[fge]

OK, so, with additionalProperties being false, this is the schema:

https://gist.github.com/fge/5037784

The "patch"/"with" combination is an invention of mine, where "patch" is a schema, and "with" is a JSON Patch. The result should replace the original, and we obtain a schema for a dog and a cat respectively.

With additional properties allowed (but ignored), the schema becomes much more simple and does not require any extension:

https://gist.github.com/fge/5037801

[fge]

OK, here is a crude example:

fge/json-schema-processor-examples@1c1ce01

It works, although the generated source leaves me somewhat perplexed. @joelittlejohn, what am I doing wrong here?

[fge]

OK, I understand: additionalProperties was not set to false. Setting it to false correctly produces a class with only the foo property...

[joelittlejohn]

Sorry, I've been a little busy today. Hopefully all is clear now.

Re supporting @JsonTypeInfo and @JsonSubTypes, this is a non-trivial task but probably possible. In general I've tried to avoid constructs that have a requirement to change the JSON data being bound (e.g. by inserting "type" into the data).

To support this, we'd need to allow the type property name (in this case "type") to be supplied as a configuration argument. In interested to know though how you would expect to supply the mapping between schema and type value. As far as I understand it, the schema attached to issue #36 is not quite right as we have these extra schemas attached under arbitrary properties in the root of of the document. It's valid because you're free to include additional properties but they have no impact.

So, how do we map type values to schemas? Maybe some configuration that allows a mapping from schema id to type value... or maybe I'm missing some easy solution here :)

[fge]

@joelittlejohn I guess it will not be that easy, but meanwhile, I have another question: where is the source file from your demo site? Maybe that will give me some hints.

I have made a demo page of your project here:

http://json-schema-validator.herokuapp.com/schema2pojo.html

But it is of course very primitive compared to your site. When I get the code of the site, I may be able to come up with something...

[joelittlejohn]

Francis,

I'm not quite sure what your goal is here. Are you trying to simply
replicate a site that already exists? Maybe you're planning to add some
additional feature? It might be simpler to work this in to
jsonschema2pojo.org.

Anyway, feel free to browse the code of the site. It's available here:

https://code.google.com/p/jsonschema2pojo/source/checkout?repo=web

It's also written in Clojure, but I think it should be obvious what Java
methods are being used.
On 26 Feb 2013 20:29, "Francis Galiegue" [email protected] wrote:

@joelittlejohn https://github.com/joelittlejohn I guess it will not be
that easy, but meanwhile, I have another question: where is the source file
from your demo site? Maybe that will give me some hints.

I have made a demo page of your project here:

http://json-schema-validator.herokuapp.com/schema2pojo.html

But it is of course very primitive compared to your site. When I get the
code of the site, I may be able to come up with something...

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/32#issuecomment-14137794
.

[fge]

@joelittlejohn

I'm not quite sure what your goal is here. Are you trying to simply
replicate a site that already exists? Maybe you're planning to add some
additional feature?

It has one additional feature already: it fully analyzes the schema for you before calling jsonschema2pojo ;) And since this is a processor, it means you can integrate it in a processing chain. For instance, chain that processor with the compiling processor (which I have already written) and generate an actual class. Possibilities are endless.

Thanks for the link to the code source of the site! I'll have a look at it and see how you pass options etc.

[joelittlejohn]

This sounds like a useful feature for the site. I've also implemented
compilation for my integration tests. I might add another button to the
site so that the actions are "Preview", "Source Jar", "Jar" where jar
delivers compiled classes.

Cheers
On 26 Feb 2013 20:53, "Francis Galiegue" [email protected] wrote:

@joelittlejohn https://github.com/joelittlejohn

I'm not quite sure what your goal is here. Are you trying to simply
replicate a site that already exists? Maybe you're planning to add some
additional feature?

It has one additional feature already: it fully analyzes the schema for
you before calling jsonschema2pojo ;) And since this is a processor, it
means you can integrate it in a processing chain. For instance, chain
that processor with the compiling processor (which I have already written)
and generate an actual class. Possibilities are endless.

Thanks for the link to the code source of the site! I'll have a look at it
and see how you pass options etc.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/32#issuecomment-14138984
.

[joelittlejohn]

But yes I see what you mean now Francis. I wasn't aware of your 'processor'
construct and how these are chained together.

Please feel free to give me a shout if you have any more questions.
On 26 Feb 2013 23:53, "Joe Littlejohn" jo************om wrote:

This sounds like a useful feature for the site. I've also implemented
compilation for my integration tests. I might add another button to the
site so that the actions are "Preview", "Source Jar", "Jar" where jar
delivers compiled classes.

Cheers
On 26 Feb 2013 20:53, "Francis Galiegue" [email protected] wrote:

@joelittlejohn https://github.com/joelittlejohn

I'm not quite sure what your goal is here. Are you trying to simply
replicate a site that already exists? Maybe you're planning to add some
additional feature?

It has one additional feature already: it fully analyzes the schema for
you before calling jsonschema2pojo ;) And since this is a processor, it
means you can integrate it in a processing chain. For instance, chain
that processor with the compiling processor (which I have already written)
and generate an actual class. Possibilities are endless.

Thanks for the link to the code source of the site! I'll have a look at
it and see how you pass options etc.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/32#issuecomment-14138984
.

[alexyiu]

Hi, Francis,

Sorry for re-activating this thread:
I have requirement of traversing a JSON Schema, similar to timolson's requirement.

About a year ago, you posted:

Well, with the new core I have created, you have the option to somehow visit a schema with
your own processor by using the .append() and .setPointer() operations on SchemaTree. This
is how, for instance, I validate syntax and keywords such as allOf or anyOf. I could write a small,
demo processor as an example and extract, for instance, all example keywords, print where
they are found and their contents.

I also looked at "SchemaTree" Object (2.0.* release).
Data stored inside "SchemaTree" look quite raw without the usage "append()" and etc operations.
Our JSON-Schema are also using "allOf" + "$ref" quite extensively.
Can you provide some examples of resolving and loading "$ref" under a "SchemaTree" object?

Pretty much, my short-term end-goals is to get the consolidated list of all properties referenced through "$ref" and "allOf" under a JSON Schema type.

Worst case scenario is: I would need to write my schemaTree walker + "$ref" loading logic. Potentially duplicated quite a signficant amount of work already done in this "json-schema-validator" package.

Thanks!

Regards,
Alex Yiu

[fge]

OK, covered by jsonschema2pojo, and since then the landscape has evolved.