jimblackler / jsonschemafriend Goto Github PK

Hello,

Is it possible to add proxy support to SchemaStore ?

Thx.

Given the schema:

{
	"$defs":{ "myType": {"type": "string"}},  
	"properties":{"myProperty": {"$ref": "#/$defs/MyType"}}  
}

with the wrong pointer to the type of 'myProperty' (for example, when type is misspelled). This schema succesfully pass the next json:

{ "myProperty": 0}

I expect that library should fail when I load the schema (with the GeneraionException or StandardGenerationException).
Currently the library passes such verification with a warning: WARNING: No match for #/$defs/MyType. But I cannot get 'programmatic' access to this case, so all warnings just leave in logs

Reason:
As I understand the standard, we cannot define the ref "#/$defs/myType2" in some another file and should be resolved in current context. So all local refrences must be available at schema resolution stage

Such behavior is critical in my application, where the library is used for schema linting as well. Will be good to have errors during schema load (for local refrences) of when schema is validated (for absolute refrences in external files).

Loading a JSON schema through a SchemaStore object results in an infinite loop if a JSON schema has a $schema that has no content (such as http://json-schema.org/schema, which returns a 301) and also does not have an $id.

Here's how this happens:

1. We enter a while (true) loop here.
2. Then we try getting content from the $schema here and content is null.
3. The store call here throws an exception due to the null content.
4. We catch this exception in the catch (JsonProcessingException e) block.
5. Then we make a Document object called doc and an Elements object called links here. However, since content is null, doc is null and links has a size of zero.
6. As a result, we never set resolved to true here because there are no links to loop through, and we display the "Was not valid JSON" warning from this block.
7. Then we continue here, loop again, hit the same problem, and keep looping ad infinitum.

Even though the jsonschemafriend readme lists a specific set of schemas that are supported in the $schema field, jsonschemafriend should throw an exception if the $schema has no content rather than looping infinitely.

Let me know if you'd like me to fix this myself or if someone closer to jsonschema friend would prefer to address this.

Thank you!

Hello @jimblackler ,
You have done a nice job especially in handling all the nested exception messages from oneOf,anyOf etc., and summarising the reason of failure in single, simple message.
Can you please provide support to customise that error message from schema itself similar to this

Hi,

Out of curiosity, since the documentation states that parsing a JSON Schema is supported: Does this tool include a feature that allows simplification of a given JSON Schema? With simplification, I mean merging all allOf constraints (and possibly de-referencing any schema references). I know that some other software tools exist that claim to support that kind of simplification, and that this is a non-trivial task. I am just interested to know if jsonschemafriend has something in this area.

Background: We have a schema that makes extensive use of allOf, but some (non-validator) tools which we would like to feed that schema do not support allOf.

Cheers,
Johannes

For the past 3 days we have been getting this error

we are using
sonGeneratorVersion = "0.4.3"
jsonSchemaFriendVersion = "0.9.1"

Thanks for the help in advance!

The JavaDoc link in Readme.md seems to be broken:

Javadocs can be found here.

Tag or commit '0.11.3' not found. Rechecking.

When loading the following schema I get a warning:

Aug 06, 2021 12:31:26 PM net.jimblackler.jsonschemafriend.Schema <init>
WARNING: No match for #/$defs/reference

{
  "$schema": "https://json-schema.org/draft/2019-09/schema",
  "$id": "urn:defs-test",
  "title": "TEST",
  "type": "object",
  "properties": {
    "testReference": {
      "$ref": "#/$defs/reference",
    }
  },
  "$defs": {
    "reference": {
      "type": "string"
    }
  }
}

Hi.
Your schema validator is just fine, but i get an error while validating base64 encoded data.

Part of the schema is:
...
"signature": {
"type": "string",
"contentEncoding": "base64"
},
...

The JSON data is:
"signature": "cckZGzGyCvsX0RRKqc94VuFlX83/XLoZjvXMT0beE79PwVsBwjwyCYJqPEPZnz0RDvl0nnV3A5+T0GCZcJAsCuZUAWhvDz6ezHkU2eBGcdwRzoL7FdaVcGeVZSJeuzkl"

The exception is:
.../signature with "Content encoding failed on: base64. Reason: Illegal base64 character 2f"]

I have no idea why the JSON data is wrong. In my opinion it is a valid base64 string.

Do you have any idea?

Greetings from germany
Stefan

It seems like a downloaded schema is saved in the system temp folder as defined in the CacheLoader class:

https://github.com/jimblackler/jsonschemafriend/blob/master/library/src/main/java/net/jimblackler/jsonschemafriend/CacheLoader.java#L19

I think it would be great if the user can choose whether to cache or not, based on whether the schema changes frequently. For example, it could be done using java property. I can submit a PR

Using SchemaStore::loadSchema(Object document) get stuck in an infinite loop trying to fetch the meta-schema of draft-07.
Getting logs:

net.jimblackler.jsonschemafriend.SchemaStore: Was not valid JSON: http://json-schema.org/draft-07/schema

It began today (2023-10-02) and probably is related to a change made at https://json-schema.org/blog/posts/new-website.

Issue

• It seems that the issue is with the http stream not following 301 redirects.
• Also getting into an infinite loop instead of failing fast makes the service unresponsive.

Reproduce

• I think any piece of code (like the one in the README.md) which loads draft-07 schema will fail.

Workaround

Workaround 1:

Using a url rewriter (a bit dangerous)

public class SchemaForceHttpsUrlRewriter implements UrlRewriter {
    @Override
    public URI rewrite(URI in) {
        if (in.toString().startsWith("http:")) {
            return URI.create(in.toString().replaceFirst("http:", "https:"));
        }
        return in;
    }
}

Initialize store with store = new SchemaStore(new SchemaForceHttpsUrlRewriter());

Workaround 2:

not using a validator when using loadSchema

store.loadSchema(obj, null);

Workaround 3:

Storing the meta schemas as resources in the module and pre-loading them before calling loadSchema
Using SchemaStore::store(URI uri, Object document)
or SchemaStore::register(URI path, Schema schema)

(Maybe as a premanent solution it would be best to bundle all meta-schema draft-03+ with the library and prevent even trying to download them)

Hi, would it be possible to add a way to make custom format validators like this? https://github.com/everit-org/json-schema#example. We are trying to migrate to your library, but this is our only real blocking point.

Thanks in advance
-Nicolas

At the moment the SchemaStore will, if it finds a schema that hasn't yet been resolved, call the getContent method to retrieve it:
https://github.com/jimblackler/jsonschemafriend/blob/master/library/src/main/java/net/jimblackler/jsonschemafriend/SchemaStore.java#L144
That getContent method will then use the static CacheLoader.load method.

I see you can also preload the SchemaStore using the public store or register methods.

What I'm asking for though is the ability to intercept that getContent loading method (in my case I may want to store schemas in some persistence layer), via say an interface ContentLoader ...

  public SchemaStore(UrlRewriter urlRewriter, boolean cacheSchema, ContentLoader contentLoader) {
     ...
     this.contentLoader = contentLoader;
  }

  private String getContent(URI documentUri) throws IOException {
    Uri uriToLoad = urlRewriter == null : documentUri : urlRewriter.rewrite(documentUri);
    if (contentLoader != null) {
      return contentLoader.load(uriToLoad, cacheSchema);
    }
    return load(uriToLoad, cacheSchema);
  }

Maybe that would also help with developers who have issues like #3 and need to provide their own means of resolving a schema?

At the moment, as no module name is specified by the project, if a user is using JPMS (jigsaw), then the the automatic module names for the core and extra jars are simply core and extra. This isn't very descriptive and its very possible there's another core jar on the module path, causing a module name clash.

Ideally, we should add a module-info class to the projects that defines the module names and which packages they export.

Until this is done it would be great if we could at least set a more descriptive / unique automatic module name in the jar manifests.

See http://branchandbound.net/blog/java/2017/12/automatic-module-name/ for more info.

Hi, would it be possible to replace CachedRegExPatternSupplier.map by a synchronized map? Or define custom supplier implements RegExPatternSupplier is recommended？

import java.util.WeakHashMap;

public class CachedRegExPatternSupplier implements RegExPatternSupplier {
  private final RegExPatternSupplier wrapped;
  private final WeakHashMap<String, RegExPattern> map = new WeakHashMap<>();

  public CachedRegExPatternSupplier(RegExPatternSupplier wrapped) {
    this.wrapped = wrapped;
  }

  @Override
  public RegExPattern newPattern(String pattern) throws InvalidRegexException {
    RegExPattern regExPattern1 = map.get(pattern);
    if (regExPattern1 == null) {
      regExPattern1 = wrapped.newPattern(pattern);
      map.put(pattern, regExPattern1);
    }
    return regExPattern1;
  }
}

Thanks!

I've prepared test suite based on https://json-schema.org/draft/2020-12/json-schema-core.html#recursive-example in json-schema-org/JSON-Schema-Test-Suite#521

Currently it fails with NullPointerException:

java.lang.NullPointerException
	at net.jimblackler.jsonschemafriend.Validator.validate(Validator.java:95)
	at net.jimblackler.jsonschemafriend.Validator.validate(Validator.java:179)
	at net.jimblackler.jsonschemafriend.Validator.validate(Validator.java:76)
	at net.jimblackler.jsonschemafriend.Validator.validate(Validator.java:412)
	at net.jimblackler.jsonschemafriend.Validator.validate(Validator.java:76)
	at net.jimblackler.jsonschemafriend.Validator.validate(Validator.java:505)
	at net.jimblackler.jsonschemafriend.Validator.validate(Validator.java:156)
	at net.jimblackler.jsonschemafriend.Validator.validate(Validator.java:71)
	at net.jimblackler.jsonschemafriend.SuiteTest.lambda$scan$1(SuiteTest.java:100)

https://advisory.checkmarx.net/advisory/vulnerability/Cx78f40514-81ff/

See https://json-schema.org/draft/2020-12/release-notes.html

Many thanks for open sourcing this great library! Would you consider uploading it to Maven Central? I can't add jitpack as a maven repository in some of the projects I'd like to use this lib, unfortunately.

Thank you very much for making this library available as open source. This is extremely useful and I am glad I chanced upon it before giving up. I am using this library to build a quality check tool that scans JSON schemas for compliance with certain design standards. There is one particular design pattern (composition) that seems to be slipping through the parser. Here is the schema snippet:
"BaseAddress": {
"type": "object",
"properties": {
"addressLineText": {
"$ref": "#/definitions/String50"
},
"addressUnitIdentifier": {
"$ref": "#/definitions/String11"
},
"cityName": {
"$ref": "#/definitions/String35"
},
"postalCode": {
"$ref": "#/definitions/PostalCodeType"
}
}
},
"ResidenceAddress": {
"allOf": [
{
"$ref": "#/definitions/BaseAddress"
},
{
"properties": {
"stateCode": {
"$ref": "#/definitions/ResidenceStateCodeType"
}
}
}
],
"required": [ "addressLineText", "cityName", "postalCode", "stateCode" ]
},
While the BaseAddress object is being correctly read, the ResidenceAddress object is not being recognized. What am I missing?

Regards

Hi , i tried tried the following .. in your online version

SChema:

"$schema": "https://json-schema.org/draft/2020-12/schema",

    "Erstellt": {
        "type": "string",
        "format": "date-time"
    },

Json:
"Erstellt": "2004-06-14T23:34:30.322",

So here the timezone is missing in the json
https://json-schema.org/understanding-json-schema/reference/string.html says in the sample that there has to be a timezone

https://github.com/networknt/json-schema-validator this lib marks that as invalid .. your online version says that it is valid ..
sry im not that deeply in the spec . but ... is there a bug in your implementation or is the spec not that precise?
Greets André

from the demo code in readme file like this:

package xxxxxx;


import com.fasterxml.jackson.databind.ObjectMapper;
import net.jimblackler.jsonschemafriend.*;

import java.util.HashMap;
import java.util.Map;

public class Test {
    static String schemaStr = "{\n" +
            "  \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n" +
            "  \"properties\": {\n" +
            "    \"name\": {\n" +
            "      \"type\": \"string\",\n" +
            "      \"minLength\": 2\n" +
            "    }\n" +
            "  }\n" +
            "}";

    static String data1 = "{\n" +
            "  \"name\": \"Bill\"\n" +
            "}\n";
    static String data2 = "{\n" +
            "  \"name\": \"\"\n" +
            "}\n";

    public static void main(String[] args) {
        try {
            ObjectMapper mapper = new ObjectMapper();
            SchemaStore schemaStore = new SchemaStore(); // Initialize a SchemaStore.

            Map map = mapper.readValue(schemaStr, HashMap.class);
            Schema schema = schemaStore.loadSchema(map);
            Validator validator = new Validator();

            // Will not throw an exception.
            validator.validate(schema, data1);
            // Will throw a ValidationException.
            validator.validate(schema, data2);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

how to work with json string.

Hi,

I am just thinking that it could be useful to have getMessage() publicly accessible: https://github.com/jimblackler/jsonschemafriend/blob/master/library/src/main/java/net/jimblackler/jsonschemafriend/ValidationError.java#L51

I just had a case today where the message need to be customized and I had to use reflection to obtain the result of getMessage()

Specifications (since draft-6) say (https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-01#name-mathematical-integers):
For consistency, integer JSON numbers SHOULD NOT be encoded with a fractional part. (should not == not recommended)
But both the properties in json below matches to {"type": "integer"}

{ "a": 5,
  "b": 5.0
}

while the only integer property is the "a"
Note thal library works in expected way when I select draft-3 or draft-4 here: https://tryjsonschematypes.appspot.com/#validate

I can load a schema from a file just fine, either as a resource, or directly using a java.io.file.
When I try to take a string, and transform it into a ByteArrayInputStream, I get this rather cryptic error.

    net.jimblackler.jsonschemafriend.StandardGenerationException: {valid=false, errors=[{error=Unexpected type in data: ByteArrayInputStream, keywordLocation=http://json-schema.org/draft-04/schema, absoluteKeywordLocation=http://json-schema.org/draft-04/schema, instanceLocation=#}]}

I checked out the code, walked through SchemaStore, added some logging, re-compiled, and re-ran and I don't see a clear path on how to fix my issue.

My use case is pretty simple. I have a string that contains a schema, and a string that contains JSON data that I want to validate with the schema mentioned earlier. I can't figure out how to do this with this library, so I am moving on to something else.

It would be useful to be able to support custom Java types when passing document to be validated via a map.

Currently, you get a UnexpectedTypeError if the input map structure contains anything else than Map, List, Number, String, Boolean or null.

I propose that you can register custom Java types and map them to a $ref in the schema.

Ideally, the validator should check for custom types first, so that you can have custom handling of e.g. java.math.BigInteger or java.math.BigDecimal (which are Numbers), and retain the standard handling of all other Numbers.

Fatal Exception: java.lang.NoClassDefFoundError: Failed resolution of: Ljava/nio/file/FileSystems;
       at net.jimblackler.jsonschemafriend.CacheLoader.<clinit>(CacheLoader.java:17)
       at net.jimblackler.jsonschemafriend.SchemaStore.<init>(SchemaStore.java:71)
       at net.jimblackler.jsonschemafriend.SchemaStore.<init>(SchemaStore.java:51)

since FileSystems is added in API version 26, it will crash on android version under 26

As you may know the package javax.mail has not been updated for a while (Maven Central shows last release in 2018), more and more projects are migrating over to Jakarta.
In FormatChecker an AddressException from javax.mail is being used, which is causing problems if software already has moved on to use Jakarta. The following exception is being thrown:
Exception in thread "main" java.lang.NoClassDefFoundError: javax/mail/internet/AddressException
It would be appreciated, if you could migrate onto newer library.

Hi,

I've recently needed to compare the performance and functionality of this and other JVM based JSON validation libraries, and I thought you might like to see how this implementation compares. I've published to the code and the results here: https://github.com/creek-service/json-schema-validation-comparison.

While I've spent time trying to get the best out of each library under test, it's possible that the results for this implementation could be improved if someone with more in-depth knowledge were to look into it. If you feel I'm not showing in its best light, please feel free to raise PRs to fix issues and improve your score.

Please feel free to link to the code and results.

I hope this is of some use to you.

Thanks,

Andy

hi,

currently, me and my teams are working with json schema to build a dynamic form for our client, and we found this library and use it as a validator.

all features work as expected and no issues on the functionality, but we found that our app size is increased by approximately 10MB and after we check it, we found that com.ibm.icu gives a contribution to the app size.

after doing some research, we found that com.ibm.icu is from this library.

can you use another library to reduce the size? that would be so much help, and I will be very appreciate that.

thank you.

When using draft/2019-09 as the schema, having a property with type string and format uuid validates that the string value is actually a valid UUID in version 0.10.6, but not in version 0.11.0 or later.

Was this actually removed from the standard in that edition or was a bug introduced in the library?

The following test succeeds when using version 0.10.6 but fails with version 0.11.0 or later:

/test-schemas/TEST.json:

{
  "$schema": "https://json-schema.org/draft/2019-09/schema",
  "$id": "urn:test:TEST",
  "title": "TEST",
  "type": "object",
  "required": [
    "testId"
  ],
  "additionalProperties": false,
  "properties": {
    "testId": {
      "type": "string",
      "format": "uuid"
    },
    "testNumber": {
      "type": "number"
    },
    "testEnum": {
      "type": "string",
      "enum": [
        "VALID_ENUM_VALUE"
      ]
    },
    "testReference": {
      "$ref": "urn:test:TEST#/$defs/reference"
    }
  },
  "$defs": {
    "reference": {
      "type": "boolean"
    }
  }
}

JsonSchemaValidatorTest.java:

package com.princh.resource.validation;

import java.net.URL;
import java.util.HashMap;
import java.util.Map;
import net.jimblackler.jsonschemafriend.GenerationException;
import net.jimblackler.jsonschemafriend.Schema;
import net.jimblackler.jsonschemafriend.SchemaStore;
import net.jimblackler.jsonschemafriend.ValidationException;
import net.jimblackler.jsonschemafriend.Validator;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

import static org.assertj.core.api.Assertions.assertThatThrownBy;

public class JsonSchemaValidatorTest {
  private JsonSchemaValidator validator;

  @BeforeEach
  public void setUp() {
    URL schemaUrl = this.getClass().getResource("/test-schemas/TEST.json");
    validator = new JsonSchemaValidator(schemaUrl);
  }

  @Test
  public void testInvalidUuid() {
    Map<String, Object> object = new HashMap<>();
    object.put("testId", "foo");

    assertThatThrownBy(() -> {
      validator.validate(object);
    }).isInstanceOf(ValidationException.class)
      .hasMessageContaining("Not compliant with format: uuid");
  }

  public static class JsonSchemaValidator {
    private final Schema schema;

    public JsonSchemaValidator(URL schemaUrl) {
      try {
        SchemaStore schemaStore = new SchemaStore();
        schema = schemaStore.loadSchema(schemaUrl);
      } catch (GenerationException e) {
        throw new RuntimeException(e);
      }
    }

    public void validate(Map<String, Object> jsonMap) throws ValidationException {
      Validator validator = new Validator();
      validator.validate(schema, jsonMap);
    }
  }
}

Hi @jimblackler,

Thought I'd let you know that I think this library is currently mishandling schemas that have format-assertion listed in its $vocabulary.

Turns out that having "https://json-schema.org/draft/2020-12/vocab/format-assertion": false e.g.:

"$vocabulary": {
        "https://json-schema.org/draft/2020-12/vocab/core": true,
        "https://json-schema.org/draft/2020-12/vocab/format-assertion": false
    },

...doesn't turn off format assertions, as I'd first assumed. Instead the false just means parsers/validators that don't understand format-assertsions can still parse the schema, (a value of true means such implementations should error out).

SchemaFriend fails 2 of the optional/format-assertions.json tests. I'm assuming this is because it turns off format assertions when it sees the "https://json-schema.org/draft/2020-12/vocab/format-assertion": false.

Here's the slack conversation about what the false means...

Given the schema:

{ 
  "defs":{ "myType": {"type": "string"}},
  "$defs":{ "myType": {"type": "string"}},  
  "properties":{
     "propA": {"$ref": "#/defs/myType"},
     "propB": {"$ref": "#/$defs/myType"},
  }  
}

When I use this schema to validate the json:

{
    "propA": 0,
    "propB": 0
}

Then I have the only 1 error for "propB" which use $ref to the #/$defs/myType
But I expect to have the error for "propA" as well. The only defference is that $ref for propA is use json pointer to "non-standard" keyword for schema location.

Actually I cannot proove that it is definetly wrong behavior, because the official test-suite for $ref does not contain such testcase. In the suite all the refrenced schemas are contained inside "$defs", "definitions" or "properties".

Bun on another side, in the standard did not find any restriction which prohibit to use refrences outside of the standard keywords. The only requirement to $ref (in 2019+) is that refreced element must contain schema. From 2020-12: "The value of the "$ref" keyword MUST be a string which is a URI-Reference. Resolved against the current URI base, it produces the URI of the schema to apply."