Unknown keyword warning when using meta schema from file about json-schema-validator

[justin-tay]

It's not really clear what you are trying to do but the code that you posted likely doesn't do anything useful.

The first parameter to JsonMetaSchema.builder should be the IRI of the meta schema ie. the id and not the retrieval URI.

Are you trying to validate a schema using the meta schema? Or are you trying to validate some inputs using a schema?

If you want to validate inputs using a schema you would need to implement the keywords / vocabularies.

If you just want to validate a schema using a meta schema then that is possible without implementing the keywords / vocabularies but you should be providing the mapping of the id eg. https://example.org/meta/schema to the retrieval URI eg. classpath:schemas/my-schema.json.

You should use https://json-schema.org/draft/2019-09/schema instead of http://json-schema.org/draft/2019-09/schema# if you want to refer to draft 2019-09.

[khouari1]

Hi @justin-tay , thank you for your response.

I am trying to validate a schema using my meta schema, so I guess I should do your second suggestion.

How can I map my meta schema id to the retrieval URI?

[justin-tay]

You should define the mappings in the factory.

package com.networknt.schema;

import java.util.Collections;

import org.junit.jupiter.api.Test;

import com.networknt.schema.SpecVersion.VersionFlag;

public class Issue1045Test {
    @Test
    void metaSchemaValidation() {
        String instanceData = "{\r\n"
                + "    \"myType\": \"bad\"\r\n"
                + "}";
        SchemaValidatorsConfig config = new SchemaValidatorsConfig();
        config.setPathType(PathType.JSON_POINTER);
        config.setFormatAssertionsEnabled(true);
        JsonSchema schema = JsonSchemaFactory
                .getInstance(VersionFlag.V201909,
                        builder -> builder.schemaMappers(
                                schemaMappers -> schemaMappers.mappings(Collections.singletonMap("https://example.org/meta/schema", "classpath:schemas/my-schema.json"))))
                .getSchema(SchemaLocation.of("https://example.org/meta/schema"), config);
        System.out.println(schema.validate(instanceData, InputFormat.JSON, OutputFormat.HIERARCHICAL));
    }
}

[khouari1]

Thanks for that example. I am still getting the warning unknown keyword message in the log. When I remove the schema mappers line I get a FileNotFoundException so that suggests it is loading my file from the classpath correctly.
Is it possible that unless the keyword is added with the java api that it will always display that warning message?
When I use the below I don't get the message:

 JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V201909) { builder ->
            builder
                .metaSchema(
                    JsonMetaSchema.builder(
                        "https://example.org/meta/schema",
                        JsonMetaSchema.getV201909(),
                    )
                        .keyword(keyword)
                        .build()
                )
                .build()
        }

[justin-tay]

You shouldn't get an unknown keyword warning unless you are attempting to load a schema that uses your custom meta schema.

Are you trying to validate a schema using the meta schema? Or are you trying to validate some inputs using a schema? You aren't showing any code to tell.

[khouari1]

Thanks Justin. I have this code to create the factory and get a schema. In this example, as shown in my first comment, I have a myType keyword in my meta schema that I want to use.

val jsonSchemaFactory: JsonSchemaFactory =
        JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V201909) { builder ->
            builder
                .schemaMappers {
                    it.mappings(mapOf("https://example.org/meta/schema" to "classpath:schemas/my-schema.json"))
                }
                .build()
        }
        
private fun getJsonSchema(jsonSchema: String): JsonSchema {
        val config = SchemaValidatorsConfig().apply {
            isHandleNullableField = true
            isJavaSemantics = true
        }
        return jsonSchemaFactory.getSchema(jsonSchema, config)
    }

And wanting to create a schema which inherits from my meta schema, something like below:

{
  "properties": {
    "FOO": {
      "description": "Message type",
      "type": "string",
      "myType": "FOO"
    }
  },
  "description": "",
  "title": "Something",
  "$defs": {},
  "$id": "https://example.org/another-schema",
  "$schema": "https://example.org/meta/schema",
  "type": "object"
}

With the above I see the warning message in the log when this line executes jsonSchemaFactory.getSchema(jsonSchema, config)

[justin-tay]

You are attempting to load a schema that uses your custom meta schema.

If you want to validate inputs using such a schema you would need to implement the keywords / vocabularies in Java to implement the desired validation behavior.

If you don't do anything then obviously the library can't do validation on unknown keywords and would by default pass validation. The warning is there to tell you that that is probably not what you want.

See https://github.com/networknt/json-schema-validator/blob/master/doc/custom-meta-schema.md

[khouari1]

Right, ok. Is there a way to generate the meta schema as json from the Java API? I need the meta schema in a file also and ideally would then like to have the Java API as the source of truth, and generate from there. Or will I need to manually create the meta schema json file alongside the java api?

[justin-tay]

No there is no way to generate the meta schema using the Java API. They are not the same thing so there's no single source of truth. The meta schema simply describes what a valid schema looks like and lists the relevant vocabularies. The Java API is for how to locate the vocabulary, keyword and format implementations.

This is much the same as the standard dialect meta schemas so you can just refer to how the standard ones look like, eg. https://json-schema.org/draft/2020-12/schema.

It is preferable to define a custom vocabulary to hold the custom keywords that you want to create.

[khouari1]

Ah I see. Question on the vocabularies, I see in this example it is creating a vocabulary "https://www.example.com/vocab/equals" https://github.com/networknt/json-schema-validator/blob/master/doc/custom-meta-schema.md#associating-vocabularies-to-a-dialect. What would be the way someone would find out what's in that vocabulary?

[khouari1]

The reason I ask is because my custom keyword is an enum and I want to share with the API client what the possible enum values are.

[justin-tay]

See https://github.com/json-schema-org/json-schema-vocabularies

In the same way the standard spec describes what the standard keywords does you would use documentation to describe it.

[khouari1]

Ok, thank you @justin-tay, I will try this out.

[khouari1]

Really appreciate your time @justin-tay . I hope there's a buy coffee link somewhere!

I now have this as my meta schema (example-meta-schema.json):

{
  "$schema": "http://json-schema.org/draft/2019-09/schema",
  "$id": "https://example.org/schema/meta/example",
  "$vocabulary": {
    "https://json-schema.org/draft/2019-09/vocab/core": true,
    "https://json-schema.org/draft/2019-09/vocab/applicator": true,
    "https://json-schema.org/draft/2019-09/vocab/unevaluated": true,
    "https://json-schema.org/draft/2019-09/vocab/validation": true,
    "https://json-schema.org/draft/2019-09/vocab/meta-data": true,
    "https://json-schema.org/draft/2019-09/vocab/format-annotation": true,
    "https://json-schema.org/draft/2019-09/vocab/content": true,
    "https://example.org/meta/vocab/example": true
  },
  "$dynamicAnchor": "meta",
  "title": "Example JSON meta schema",
  "allOf": [
    {
      "$ref": "https://json-schema.org/draft/2019-09/schema"
    },
    {
      "$ref": "https://example.org/schema/meta/vocab/example"
    }
  ]
}

and code:

    val vocabularyFactory = VocabularyFactory { iri ->
        if ("https://example.org/meta/vocab/example" == iri) {
            Vocabulary("https://example.org/meta/vocab/example", keyword)
        } else {
            null
        }
    }

    val jsonSchemaFactory: JsonSchemaFactory =
        JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V201909) { builder ->
            builder
                .metaSchema(
                    JsonMetaSchema.builder("https://example.org/schema/meta/example", JsonMetaSchema.getV201909())
                        .vocabularyFactory(vocabularyFactory)
                        .unknownKeywordFactory(DisallowUnknownKeywordFactory())
                        .build()
                )
                .schemaMappers {
                    it.mappings(mapOf(
                        "https://example.org/schema/meta/example" to "classpath:schemas/example-meta-schema.json",
                    ))
                }
                .build()
        }

Input is something like:

{
  "properties": {
    "FOO": {
      "description": "Message type",
      "type": "string",
      "myType": "FOO"
    }
  },
  "description": "",
  "title": "Something",
  "$defs": {},
  "$id": "https://example.org/another-schema",
  "$schema": "https://example.org/schema/meta/example",
  "type": "object"
}

I am still getting unknown keyword error (I added DisallowUnknownKeywordFactory to help with debugging). Is the above how I am supposed to be doing this?

[justin-tay]

You cannot convert a 2020-12 meta schema into a 2019-09 meta schema just by changing 2020-12 to 2019-09. The keywords aren't the same, for instance $dynamicAnchor is not supported in 2019-09. In 2019-09 they used $recursiveAnchor. Also the vocabularies are different. 2019 doesn't use format-annotation. See https://json-schema.org/draft/2019-09/schema.

The vocabularies are typically associated with the dialect not the actual custom meta schema so that it can load other custom meta schemas that use the vocabulary without explicit configuration. I.e. as per the docs you should configure it on 2019-09 and not https://example.org/schema/meta/example.

That said I don't see anything other thing obviously wrong so you're going to have to set breakpoints and debug.

[justin-tay]

json-schema-validator/src/test/java/com/networknt/schema/VocabularyTest.java

Lines 171 to 217 in 4045e7e

	@Test
	void customVocabulary() {
	String metaSchemaData = "{\r\n"
	+ " \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\r\n"
	+ " \"$id\": \"https://www.example.com/no-validation-no-format/schema\",\r\n"
	+ " \"$vocabulary\": {\r\n"
	+ " \"https://www.example.com/vocab/format\": true,\r\n"
	+ " \"https://json-schema.org/draft/2020-12/vocab/applicator\": true,\r\n"
	+ " \"https://json-schema.org/draft/2020-12/vocab/core\": true\r\n"
	+ " },\r\n"
	+ " \"allOf\": [\r\n"
	+ " { \"$ref\": \"https://json-schema.org/draft/2020-12/meta/applicator\" },\r\n"
	+ " { \"$ref\": \"https://json-schema.org/draft/2020-12/meta/core\" }\r\n"
	+ " ]\r\n"
	+ "}";
	String schemaData = "{\r\n"
	+ " \"$id\": \"https://schema/using/no/format\",\r\n"
	+ " \"$schema\": \"https://www.example.com/no-validation-no-format/schema\",\r\n"
	+ " \"hello\": {\r\n"
	+ " \"dateProperty\": {\r\n"
	+ " \"format\": \"date\"\r\n"
	+ " }\r\n"
	+ " }\r\n"
	+ "}";
	VocabularyFactory vocabularyFactory = uri -> {
	if ("https://www.example.com/vocab/format".equals(uri)) {
	return new Vocabulary("https://www.example.com/vocab/format", new AnnotationKeyword("hello"));
	}
	return null;
	};
	JsonMetaSchema metaSchema = JsonMetaSchema
	.builder(JsonMetaSchema.getV202012().getIri(), JsonMetaSchema.getV202012())
	.vocabularyFactory(vocabularyFactory)
	.build();
	JsonSchemaFactory factory = JsonSchemaFactory
	.getInstance(VersionFlag.V202012,
	builder -> builder.metaSchema(metaSchema).schemaLoaders(schemaLoaders -> schemaLoaders.schemas(Collections
	.singletonMap("https://www.example.com/no-validation-no-format/schema",
	metaSchemaData))));
	JsonSchema schema = factory.getSchema(schemaData);
	OutputUnit outputUnit = schema.validate("{}", InputFormat.JSON, OutputFormat.HIERARCHICAL, executionContext -> {
	executionContext.getExecutionConfig().setAnnotationCollectionEnabled(true);
	executionContext.getExecutionConfig().setAnnotationCollectionFilter(keyword -> true);
	});
	assertNotNull(outputUnit.getAnnotations().get("hello"));
	}

[khouari1]

Hi @justin-tay , I believe I have it working now. Thank you so much for your help!