Comments (6)
I attach the problematic schema here too, for convenience.
schema.json
from json-schema-validator.
Maybe it'll be of help, bisect shows this commit as the one after which the code hangs: 7f1ec11 (#931).
from json-schema-validator.
Unfortunately this typically means that there's not enough memory to load your schema and you would need to allocate more heap for your JVM.
Previously the default was to lazily load the schema. Now the default is to preload the schema.
You can still configure it to lazily load the schema with the setPreloadJsonSchema
option.
@Test
void testNoErrorsForEmptyObject() throws IOException {
SchemaValidatorsConfig config = new SchemaValidatorsConfig();
config.setPreloadJsonSchema(false);
getJsonSchemaFromClasspath(resource("schema.json"), SpecVersion.VersionFlag.V7, config);
}
However, you should note that if you have inputs that exercise all the possible evaluation paths of your schema as defined, then it's just going to OOM later when if validates such an input.
from json-schema-validator.
That seems... improbable, @justin-tay ? I mean - this schema is not that large (or complex). Something seems wrong if it needs in excess of gigabytes of memory to load validators for a mere 300kb?
from json-schema-validator.
The state of the validators / schemas are different along the evaluation path. Therefore the code will need to create validator / schema instances at each evaluation path to store this state. This is cached since throwing it away impacts performance as this state is fixed for a particular schema. Throwing it away means that the same state needs to be regenerated between each run and also increases the objects that need to be garbage collected.
The following is a small sample of the evaluation paths that get loaded for your schema.
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/2/$ref/properties/entries/oneOf/1/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/2/$ref/properties/comment/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/3/$ref/properties/dictionary/oneOf/1/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/3/$ref/properties/comment/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/4/$ref/properties/comment/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/5/$ref/properties/comment/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/3/$ref/properties/labelAggregator/allOf/0/$ref/oneOf/0/$ref/properties/labelCollector/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/3/$ref/properties/labelAggregator/allOf/0/$ref/oneOf/0/$ref/properties/labelCollector/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/0/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/3/$ref/properties/labelAggregator/allOf/0/$ref/oneOf/0/$ref/properties/labelCollector/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/1/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/3/$ref/properties/labelAggregator/allOf/0/$ref/oneOf/0/$ref/properties/labelCollector/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/1/$ref/properties/operator/oneOf/1/$ref
/properties/stages/additionalProperties/$ref/oneOf/5/$ref/properties/clusters/allOf/0/$ref/oneOf/1/$ref/properties/matrix/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/2/$ref/properties/vectors/properties/columns/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/11/$ref/properties/documentPairs/allOf/0/$ref/oneOf/1/$ref/properties/validation/allOf/0/$ref/properties/pairwiseSimilarity/allOf/0/$ref/oneOf/1/$ref/properties/features/allOf/0/$ref/oneOf/9/$ref/properties/minFeatureCount/oneOf/1/$ref
/properties/stages/additionalProperties/$ref/oneOf/5/$ref/properties/clusters/allOf/0/$ref/oneOf/1/$ref/properties/matrix/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/2/$ref/properties/vectors/properties/columns/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/11/$ref/properties/documentPairs/allOf/0/$ref/oneOf/1/$ref/properties/validation/allOf/0/$ref/properties/pairwiseSimilarity/allOf/0/$ref/oneOf/1/$ref/properties/features/allOf/0/$ref/oneOf/9/$ref/properties/maxFeatureCount/oneOf/0/allOf/0/$ref
/properties/stages/additionalProperties/$ref/oneOf/5/$ref/properties/clusters/allOf/0/$ref/oneOf/1/$ref/properties/matrix/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/2/$ref/properties/vectors/properties/columns/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/11/$ref/properties/documentPairs/allOf/0/$ref/oneOf/1/$ref/properties/validation/allOf/0/$ref/properties/pairwiseSimilarity/allOf/0/$ref/oneOf/1/$ref/properties/features/allOf/0/$ref/oneOf/9/$ref/properties/maxFeatureCount/oneOf/1/$ref
/properties/stages/additionalProperties/$ref/oneOf/5/$ref/properties/clusters/allOf/0/$ref/oneOf/1/$ref/properties/matrix/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/2/$ref/properties/vectors/properties/columns/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/11/$ref/properties/documentPairs/allOf/0/$ref/oneOf/1/$ref/properties/validation/allOf/0/$ref/properties/pairwiseSimilarity/allOf/0/$ref/oneOf/1/$ref/properties/features/allOf/0/$ref/oneOf/9/$ref/properties/comment/$ref
Looking at this it looks like even if you don't preload, you could still potentially get an OOM during execution if you have inputs that exercise the evaluation paths and if you don't want to allocate memory for all the evaluation paths as there is currently no option not to cache the loaded schema during the run itself.
This probably needs more memory than the consolidated FHIR schema, which like a 4 MB schema so it's not really the size in bytes of the schema itself.
from json-schema-validator.
Right, I see it now. It's the fanout on evaluation of refs then - you can easily make this exponential and crash schema parsing even with a very small input. It's basically what's happening with the example I attached. Thank you for looking for a workaround - I don't think multiple refs are going to be that uncommon in the wild, so it's probably good to have an option to turn off caching.
from json-schema-validator.
Related Issues (20)
- the errors is garbled characters HOT 1
- Keyword Walker not walking root of then/else HOT 1
- JSON Schema validation failed against custom JSON Meta Schema after update 1.2.0 -> 1.3.0 HOT 2
- 1.4.0: Is there a way to perform local verification? HOT 5
- Bidi rule validation implementation does not match the specification HOT 3
- Additional i18n language(spanish) support related (#847) HOT 8
- We Need Additional Spanish Language support can you please add the spanish langauge our customers need it we already in PROD HOT 7
- Schema from String HOT 1
- change example.com connection error IOException
- improve outputUnits with all assertion fields HOT 3
- V1.4.0 ValidatorTypeCode.getKeywords(SpecVersion.VersionFlag.V201909) has missing keywords HOT 2
- Error occurs if OneOf and AllOf are nested HOT 10
- Do the values of allOf need to be validated as arrays?
- Add an option to toggle debug message HOT 1
- Impossible to construct a SchemaLocation containing a json pointer with an escaped fragment
- Impossible to construct a SchemaLocation containing a json pointer with an escaped fragment HOT 1
- EmailValidator limits the user length to be <= 64 HOT 2
- JsonSchemaFactory.builder().defaultMetaSchemaIri() and new Version4().getInstance().getIri()
- Question about JSON validation: Type "object" can be null? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from json-schema-validator.