Offline validation about json-schema-validator

[fge]

Hello, and thank you for your interest in this library.

Would you mind tell me what version you are using? 0.4.x and 0.5.0betaX have diverged quite a lot.

[fge]

OK, I guess you are using the current HEAD or 0.5.0beta3.

It is the SchemaContainer constructor which gets in the way: it refuses to build a container if the id of the schema is not absolute.

This is a design choice from my part. The problem is that a non absolute URI means the schema cannot be reliably addressed from outside itself. Another choice would be to just scrap the id and return an anonymous schema container instead.

Can you provide some sample data?

[gitgrimbo]

Hi, yes I'm using 0.5.0beta3 from Maven central.

Here's a simple example.

{
    "type": "object",
    "$schema": "http://json-schema.org/draft-03/schema",
    "id": "Request",
    "properties": {
        "requestName": {
            "type": "string",
            "required": true,
            "description": "The request name."
        }
    }
}

Would this section of http://tools.ietf.org/html/draft-zyp-json-schema-03#section-5.27 demonstrate that relative URIs are allowed? My emphasis below.

5.27. id

This attribute defines the current URI of this schema (this attribute
is effectively a "self" link). This URI MAY be relative or absolute.
If the URI is relative it is resolved against the current URI of the
parent schema it is contained in. If this schema is not contained in
any parent schema, the current URI of the parent schema is held to be
the URI under which this schema was addressed. If id is missing, the
current URI of a schema is defined to be that of the parent schema.
The current URI of the schema is also used to construct relative
references such as for $ref.

So does this mean that the the current URI of the parent schema is held to be "http://mysite.com/schema/Request", if the schema is hosted from that URI?

Thanks.

[fge]

The wording in this section has always been a sore point to me, and leaves a LOT to be desired. There are several problems with it:

• if "id" is relative, once you get out of the schema, you cannot reference it anymore, the draft does not cover this situation at all;
• even if you create an absolute root of yours, chances are non zero that another schema somewhere will try and register with that same root -- boom.

Since commit 0e0b4f4, I have changed the behaviour as I described in my previous post: an attempt to register a schema with a non absolute id yields an anonymous schema. This will make it into 0.5.0beta4.

Note that the API has changed quite a bit since 0.5.0beta3, and I have yet to document it. In the meanwhile, the shortest workaround I can see is for your schema to have no id at all... This may, or may not, be an acceptable workaround. I strongly suggest waiting for 0.5.0beta4 to "hit the shelves", the API is so much nicer there is just no comparison (and yes, it will be documented).

[gitgrimbo]

Thanks for your detailed response. I'll give the HEAD a try first, and then try with no "id" property.

[fge]

Fixed in 0.5.0beta4. Unfortunately, I forgot about your other issue :/

As I said, beware: the API has changed a lot. But it is fully documented!

When you have tested, can you confirm that it works for you?

[fge]

OK, 0.5.0beta5 is out there now which fixes both of your issues.

I'd appreciate if you could confirm in the relevant issues that you consider them resolved (and if not, why ;))

Thanks in advance!

[gitgrimbo]

Hi,

I've downloaded 0.5.0beta5, but I'm not sure it fixes my problems.

Caused by: java.lang.IllegalArgumentException: requested URI (../common/Response.json#) is not absolute
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
    at org.eel.kitchen.jsonschema.uri.URIManager.getContent(URIManager.java:114)
    at org.eel.kitchen.jsonschema.main.SchemaRegistry$1.load(SchemaRegistry.java:44)
    at org.eel.kitchen.jsonschema.main.SchemaRegistry$1.load(SchemaRegistry.java:39)

A.json

{
    "type": "object",
    "extends": {
        "$ref": "../common/Response.json"
    },
    "$schema": "http://json-schema.org/draft-03/schema",
    "id": "A",
    "properties": {
        "body": {
            "$ref": "bodies/ABody.json",
            "required": true
        }
    }
}

Response.json

{
    "type": "object",
    "$schema": "http://json-schema.org/draft-03/schema",
    "id": "Response",
    "properties": {
        "header": {
            "$ref": "Header.json",
            "required": true
        }
    }
}

I can hack around this by doing the following:

public static void main(String[] args) throws IOException {
    String root = "C:/path/to/json";
    String resource = root + "/schema/a/A.json";
    JsonSchemaFactory f = new JsonSchemaFactory.Builder().addURIDownloader("file", new URIDownloader() {
        public InputStream fetch(URI source) throws IOException {
            return source.toURL().openStream();
        }
    }).build();

    URL resourceURL = new File(resource).toURL();
    JsonNode schemaNode = JsonLoader.fromURL(resourceURL);

    String id = schemaNode.get("id").asText();
    URI uri = URI.create(id);
    if (!uri.isAbsolute()) {
        JsonNode newId = new TextNode(resourceURL.toExternalForm());
        ((ObjectNode) schemaNode).put("id", newId);
    }

    SchemaContainer sc = new SchemaContainer(schemaNode);
    JsonSchema schema = f.createSchema(sc);
    // do validation
}

basically, I register a "file" schema handler, and edit the schemaNode so that it uses an absolute file reference, before I pass it on for validation.

I then get these errors (I print out some extra detail here):

Exception in thread "main" java.lang.AssertionError: JSON validation failed.
schema=org.eel.kitchen.jsonschema.main.JsonSchema@4cbc4cbc.
schema=org.eel.kitchen.jsonschema.main.JsonSchema@4cbc4cbc. path=#. msgs=[#: missing required properties in instance, #: missing required properties in instance].

Which unfortunately don't provide the names of the missing fields, or the full id of the schema, which would be helpful here.

I can try and provide a standalone test if I get some time over the next couple of days.

Thanks.

[fge]

OK, the problem here is that the library currently has no notion of a namespace.

What happens is that if you register your schema using the .registerSchema() method from the factory, as the id of the schema is not absolute, the registered schema will be an anonymous schema, ie with "#" as an id.

During ref resolution (this happens in RefResolverJsonValidator), if a ref is encountered, is is resolved against the id -- but you are rootless, therefore this is a no-op and you find yourself with a relative URI -- and the library yells.

There is a better solution than what you actually do. Once you have registered your custom URIDownloader, you can just do:

final SchemaContainer container = factory.get("file:///whatever");

Admittedly, this is not very well documented. I should probably add some wiki pages.

Note that getting containers this way discards any id the document previously had!

[fge]

The more I think of it, the more I find the notion of namespace appealing. Consider:

final URI baseURI = "file:///c:/path/to/json";

// Note: still the user's responsibility to ensure that the URI scheme is supported
builder.setNamespace(baseURI);

factory = builder.build();

I have already included that into the roadmap.

[fge]

Well, I have implemented my own suggestion, but .setNameSpace() takes a String as an argument, not an URI. Of course, it is the caller's responsibity to:

• ensure the URI is valid,
• ensure the appropriate scheme is supported :)

[fge]

Hello,

Can I close this issue or do you still have issues wrt that point in particular?

[gitgrimbo]

Hi,

Apologies for the delay - I've had other stuff to contend with!

I've taken a look at your comments, and this is the structure I have now, which passes a simple test of my major objective of using file-system based, non-absolute schema ids. So thankyou for the advice.

Test code

public final class NonAbsoluteSchemaUris {
    JsonSchemaFactory factory;

    @BeforeTest
    public void beforeTest() {
        factory = new JsonSchemaFactory.Builder().registerScheme("file", new URIDownloader() {
            public InputStream fetch(URI source) throws IOException {
                return source.toURL().openStream();
            }
        }).build();
    }

    @Test
    public void testEmptyObject() throws Exception {
        doValidate("/grimbo/child.json", "/grimbo/empty-object.json");
    }

    @Test
    public void testTestObject() throws Exception {
        doValidate("/grimbo/child.json", "/grimbo/test-object.json");
    }

    private void doValidate(String schemaResource, String dataResource) throws Exception {
        // will get "file://" URIs for the schema and the data file.
        doValidate(getClass().getResource(schemaResource), getClass().getResource(dataResource));
    }

    private void doValidate(URL schemaResource, URL dataResource) throws Exception {
        JsonNode data = JsonLoader.fromURL(dataResource);

        final SchemaContainer container = factory.getSchema(schemaResource.toURI());
        final JsonSchema schema = factory.createSchema(container);

        ValidationReport r = schema.validate(data);

        System.out.println("\n----------\n");
        new ObjectMapper().writer(new DefaultPrettyPrinter()).writeValue(System.out, r.asJsonNode());
        System.out.println("\n----------\n");

        Assert.assertTrue(r.isSuccess());
    }

}

test-object.json

{
    "header": {
        "headerItem": ""
    },
    "body": {
        "bodyItem": ""
    }
}

empty-object.json

{
}

child.json

{
    "type": "object",
    "extends": {
        "$ref": "parent.json"
    },
    "$schema": "http://json-schema.org/draft-03/schema",
    "id": "child",
    "properties": {
        "body": {
            "$ref": "child-body.json",
            "required": true
        }
    }
}

parent.json

{
    "type": "object",
    "$schema": "http://json-schema.org/draft-03/schema",
    "id": "parent",
    "properties": {
        "header": {
            "$ref": "header.json",
            "required": true
        }
    }
}

header.json

{
    "type": "object",
    "$schema": "http://json-schema.org/draft-03/schema",
    "id": "header",
    "properties": {
        "headerItem": {
            "type": "string",
            "required": true
        }
    }
}

child-body.json

{
    "type": "object",
    "$schema": "http://json-schema.org/draft-03/schema",
    "id": "child-body",
    "properties": {
        "bodyItem": {
            "type": "string",
            "required": true
        }
    }
}

Error for empty-object.json

{
  "" : [ {
    "domain" : "validation",
    "keyword" : "properties",
    "message" : "required property(ies) not found",
    "missing" : [ "header" ],
    "required" : [ "header" ]
  }, {
    "domain" : "validation",
    "keyword" : "properties",
    "message" : "required property(ies) not found",
    "missing" : [ "body" ],
    "required" : [ "body" ]
  } ]
}

[gitgrimbo]

Hi,

The above test was too simple. I've forked your repo and uploaded some tests to a "uri-tests" branch.

You can see the commit here gitgrimbo@5e2fac1.

Summary

Some schemas are being resolved as relative to the 'top-most' schema, instead of resolved as relative to the schema that actually pulled them in. I.e., "/grimbo/common/header" is being resolved relative to "/grimbo/child1/child" and not relative to "/grimbo/common/parent".

As "/grimbo/child1/header" does not exist, the test reports an error.

Output

{
  "/body/header" : [ {
    "domain" : "$ref resolving",
    "keyword" : "N/A",
    "message" : "failed to get content from URI",
    "exception-class" : "org.eel.kitchen.jsonschema.main.JsonSchemaException",
    "exception-message" : "domain: $ref resolving; keyword: N/A; message: cannot fetch content from URI; uri: \"file:/C:/Users/grimbo/git/json-schema-validator/target/test-classes/grimbo/child1/header.json#\"",
    "uri" : "file:/C:/Users/grimbo/git/json-schema-validator/target/test-classes/grimbo/child1/header.json#"
  } ]
}```

[fge]

You seem to be using HEAD, by witnessing the calls to .asJsonNode().

Comments on your code:

• Natively, HEAD includes both support for the file scheme and namespace support. Which means:
  • you don't need to register your own downloader anymore,
  • you can use .setNamespace(String) to a URI against which all URIs will be resolved (by default, this URI is empty).
• You seem to have been able to use JsonSchemaException over another JsonSchemaException, I didn't expect that ;)

I'll toy with your branch and come back to you.

Thanks a lot for your input!

[fge]

I have other comments about your data. When I see your inputs, I do believe a namespace is the solution for you. Just as I suspected, your $refs are all relative to a common root: a directory on your local filesystem. The namespace, here, will be your common root.

[gitgrimbo]

Hi again. My schema paths are relative with-respect-to the schema that references them - not relative to a common root.

E.g. "/child1/child" references "../common/parent" which is "/common/parent" and ok, that exists.

Then "/common/parent" references "header" which in my mind should be "/common/header", but this is resolved as "/child1/header" which results in an error.

Maybe what I'm thinking isn't how it should work?

Thanks for your help.

[fge]

Argh.

OK, the problem is with URI resolution. If you resolve uri /x against a:///b/c/d, it will yield a:///x, as per URI resolving rules.

I don't see myself messing with such a standard mechanism too much. The real solution here is to make your ids not begin with a /, and use a namespace. This should solve your problem, I think.

[fge]

Another thing: the "top level" id of your schema is not absolute! As I already told, this means in this case the API will return an anonymous schema.

[fge]

Hello again,

I have cherry picked your test (but rewrote it, though), see https://github.com/fge/json-schema-validator/blob/master/src/test/java/grimbo/NonAbsoluteSchemaUris.java.

However, in your original tests, you expected validation of your test-object-no-bodyItem to succeed, but in fact it fails to validate: in child-body.json, it is mentioned that body must have a bodyItem property, which is not there in the test file.

I think the bug is also here in 0.5.x. If so, I'll apply the same commit -- and add your tests as well.

[fge]

0.5.x was equally affected. Now all should be fine, can you confirm?

[gitgrimbo]

Hi,

I've merged my branch with your HEAD (as of now!) and added a new test. Apologies, I'm still using my old test format (and also apologies for using my own file system paths in the tests. I'm not familiar enough with the codebase yet to know how to properly approach this).

The tests I'm interested in seem to work now so thanks for your efforts.

The new test is here, and checks to see that when you refer to a schema using an absolute URI, the relative paths in that schema are followed correctly (relative to the absolute schema).

E.g., if schema "A" has a property that $refs schema "/my/absolute/path", any relative $refs from that schema are interpreted relative to "/my/absolute/path".

gitgrimbo@986e0f8

Cheers.

[fge]

Hello,

You seem to be double testing with me:

• I do test about missing properties already (at the keyword validation level -- see PropertiesKeywordValidatorTest),
• I also do test that relative URIs are resolved wrt the current schema's URI (in the rewrite of your original test).

I had no doubt that relative URIs were followed for the simple fact that I use the draft v3 schema for validation, and its id is absolute -- which means the returned schema container will have this as a locator -- and it uses plenty of references to #. See also SchemaRegistryTest.

Here are the mechanics:

• at any point in time, a validation context has a reference to the current schema container; it is guaranteed that this schema container is in the matching schema registry (there is one per factory);
• if a JSON Reference is encountered, it is up to the JsonResolver's .resolve() method to produce a new schema node with, if the need arises, a new container;
• this new container will be set onto the context, in InstanceValidator;
• at the next round, it is this new container which will be used for ref resolving.

In fact, the core function is JsonResolver's .resolve(). And the bug you uncovered was due to the fact that the context's container was updated at places where it shouldn't have been: it needs only to be pushed to/popped from the context in InstanceValidator, which is the only entry point to the actual schema validation (see JsonResolverCache if you want to dig further).

If any of the above is not clear, please tell me. It may help me write better documentation.

Have fun!

[fge]

I actually meant JsonValidatorCache, not JsonResolverCache, sorry...

[gitgrimbo]

Hi,

Yeah I know I've been doubling-up on some tests you already have. The tests I wrote were for the particular problem I was having, and so my tunnel-vision will have led to duplicates.

You seem to have taken care of those problems so that's great.

If you want to rename / refactor the files I contributed then please do. I'd prefer you had consistency with your package names and vision for the project.

Thanks for your efforts.

[fge]

Hello,

Thanks to you for uncovering a real bug! Your test case has helped me understand where the problem came from, so I just copied over your files, I just brought them up to date with what existed in the API ;)

Just for curiosity, which version do you intend to use? 0.5.5 and 0.6.0 both fix your bug, but you seem to be using master...

[gitgrimbo]

Hi, yes I'm using master as it seems like the only way to keep up with your prolific changes!

I'm now noticing a possibly related issue. When I validate my objects against a file based schema (file://) it works.

When I validate against the same schemas packaged in a jar, I get this error. It looks like the relative paths are not being correctly assembled relative to the referer.

It seems very similar to the error I was having with file:// URIs before you fixed that issue.

I'm not sure if schema lookup is supposed to work in jars the way I'm assuming?

Schema URI

jar:file:/C:/artifacts/.m2/repository/a/b/schemas/0.0.1-SNAPSHOT/schemas-0.0.1-SNAPSHOT.jar!/schema/v1/MyResponse.json

Schema

{
  "type":"object",
  "extends":{
    "$ref":"../common/Response.json"
  },
  "$schema":"http://json-schema.org/draft-03/schema",
  "id":"MyResponse",
  "properties":{
    "body":{
      "$ref":"bodies/MyResponseResponseBody.json",
      "required":true
    }
  }
}

Error

Exception in thread "main" com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: requested URI (../common/Response.json#) is not absolute
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4884)
    at org.eel.kitchen.jsonschema.validator.JsonValidatorCache.getValidator(JsonValidatorCache.java:86)
    at org.eel.kitchen.jsonschema.validator.ValidationContext.newValidator(ValidationContext.java:105)
    at org.eel.kitchen.jsonschema.keyword.ExtendsKeywordValidator.validate(ExtendsKeywordValidator.java:72)
    at org.eel.kitchen.jsonschema.keyword.KeywordValidator.validateInstance(KeywordValidator.java:98)
    at org.eel.kitchen.jsonschema.validator.InstanceValidator.validate(InstanceValidator.java:66)
    at org.eel.kitchen.jsonschema.main.JsonSchema.validate(JsonSchema.java:67)
    at a.b.itest.JSONAssert.assertValidWithRespectToSchema(JSONAssert.java:104)
    at a.b.itest.MyResponseTempMain.main(MyResponseTempMain.java:62)
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: requested URI (../common/Response.json#) is not absolute
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
    at org.eel.kitchen.jsonschema.ref.SchemaRegistry.get(SchemaRegistry.java:113)
    at org.eel.kitchen.jsonschema.validator.JsonResolver.resolve(JsonResolver.java:123)
    at org.eel.kitchen.jsonschema.validator.JsonValidatorCache$1.load(JsonValidatorCache.java:111)
    at org.eel.kitchen.jsonschema.validator.JsonValidatorCache$1.load(JsonValidatorCache.java:104)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
    ... 12 more
Caused by: java.lang.IllegalArgumentException: requested URI (../common/Response.json#) is not absolute
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
    at org.eel.kitchen.jsonschema.uri.URIManager.getContent(URIManager.java:181)
    at org.eel.kitchen.jsonschema.ref.SchemaRegistry$1.load(SchemaRegistry.java:72)
    at org.eel.kitchen.jsonschema.ref.SchemaRegistry$1.load(SchemaRegistry.java:67)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
    ... 23 more

Let me know if I can be of more assistance!

[fge]

Hello,

I have just tested that:

JsonRef.fromString("jar:file:/C:/artifacts/.m2/repository/a/b/schemas/0.0.1-SNAPSHOT/schemas-0.0.1-SNAPSHOT.jar!/schema/v1/MyResponse.json").isAbsolute()

returns true as expected. But here you seem to be using a rootless schema. How do you load it exactly?

[gitgrimbo]

Hi,

I've uploaded a test here:

gitgrimbo@730aa75

[fge]

Can you try and use the methods provided by the factory instead?

If you have a zip file created somewhere, you're better off using the .getSchema() method from the factory and pass an absolute jar URI directly (use URI.build() to create the URI).

Admittedly, this method is misnamed, and it should have an equivalent taking a String as an argument.

[fge]

Note: in case you haven't noticed, the project has moved.

[fge]

Hello again,

I don't find this file:

"/grimbo/fake/absolute/location/child.json"

Therefore I get NPEs (expectedly). Did you forget to commit some files?

EDIT: found them

[fge]

OK, please see here:

https://github.com/json-schema/json-schema-validator/issues/1

I have found the origin of the bug.

That one is going to be a real PITA...

[fge]

OK, all fixed.

I'll close this issue since all your problems are solved, but please do see my revamped test on how to use the API: there is no need to test for missing properties, and you can use a namespace for relative URI resolution.

Also, new feature requests/issues should be opened in the new location of that project ;)