jazzband / jsonmodels

jsonmodels is library to make it easier for you to deal with structures that are converted to, or read from JSON.

JSON models

jsonmodels is library to make it easier for you to deal with structures that are converted to, or read from JSON.

• Free software: BSD license
• Documentation: http://jsonmodels.rtfd.org
• Source: https://github.com/jazzband/jsonmodels

Features

• Fully tested with Python 3.8+.
• Support for PyPy 3.9 and 3.10 (see implementation notes in docs for more details).

• Create Django-like models:

  from jsonmodels import models, fields, errors, validators
  
  
  class Cat(models.Base):
  
      name = fields.StringField(required=True)
      breed = fields.StringField()
      love_humans = fields.IntField(nullable=True)
  
  
  class Dog(models.Base):
  
      name = fields.StringField(required=True)
      age = fields.IntField()
  
  
  class Car(models.Base):
  
      registration_number = fields.StringField(required=True)
      engine_capacity = fields.FloatField()
      color = fields.StringField()
  
  
  class Person(models.Base):
  
      name = fields.StringField(required=True)
      surname = fields.StringField(required=True)
      nickname = fields.StringField(nullable=True)
      car = fields.EmbeddedField(Car)
      pets = fields.ListField([Cat, Dog], nullable=True)

• Access to values through attributes:

  >>> cat = Cat()
  >>> cat.populate(name='Garfield')
  >>> cat.name
  'Garfield'
  >>> cat.breed = 'mongrel'
  >>> cat.breed
  'mongrel'

• Validate models:

  >>> person = Person(name='Chuck', surname='Norris')
  >>> person.validate()
  None
  
  >>> dog = Dog()
  >>> dog.validate()
  *** ValidationError: Field "name" is required!

• Cast models to python struct and JSON:

  >>> cat = Cat(name='Garfield')
  >>> dog = Dog(name='Dogmeat', age=9)
  >>> car = Car(registration_number='ASDF 777', color='red')
  >>> person = Person(name='Johny', surname='Bravo', pets=[cat, dog])
  >>> person.car = car
  >>> person.to_struct()
  {
      'car': {
          'color': 'red',
          'registration_number': 'ASDF 777'
      },
      'surname': 'Bravo',
      'name': 'Johny',
      'nickname': None,
      'pets': [
          {'name': 'Garfield'},
          {'age': 9, 'name': 'Dogmeat'}
      ]
  }
  
  >>> import json
  >>> person_json = json.dumps(person.to_struct())

• You don't like to write JSON Schema? Let jsonmodels do it for you:

  >>> person = Person()
  >>> person.to_json_schema()
  {
      'additionalProperties': False,
      'required': ['surname', 'name'],
      'type': 'object',
      'properties': {
          'car': {
              'additionalProperties': False,
              'required': ['registration_number'],
              'type': 'object',
              'properties': {
                  'color': {'type': 'string'},
                  'engine_capacity': {'type': ''},
                  'registration_number': {'type': 'string'}
              }
          },
          'surname': {'type': 'string'},
          'name': {'type': 'string'},
          'nickname': {'type': ['string', 'null']}
          'pets': {
              'items': {
                  'oneOf': [
                      {
                          'additionalProperties': False,
                          'required': ['name'],
                          'type': 'object',
                          'properties': {
                              'breed': {'type': 'string'},
                              'name': {'type': 'string'}
                          }
                      },
                      {
                          'additionalProperties': False,
                          'required': ['name'],
                          'type': 'object',
                          'properties': {
                              'age': {'type': 'number'},
                              'name': {'type': 'string'}
                          }
                      },
                      {
                          'type': 'null'
                      }
                  ]
              },
              'type': 'array'
          }
      }
  }

• Validate models and use validators, that affect generated schema:

  >>> class Person(models.Base):
  ...
  ...     name = fields.StringField(
  ...         required=True,
  ...         validators=[
  ...             validators.Regex('^[A-Za-z]+$'),
  ...             validators.Length(3, 25),
  ...         ],
  ...     )
  ...     age = fields.IntField(
  ...         nullable=True,
  ...         validators=[
  ...             validators.Min(18),
  ...             validators.Max(101),
  ...         ]
  ...     )
  ...     nickname = fields.StringField(
  ...         required=True,
  ...         nullable=True
  ...     )
  ...
  
  >>> person = Person()
  >>> person.age = 11
  >>> person.validate()
  *** ValidationError: '11' is lower than minimum ('18').
  >>> person.age = None
  >>> person.validate()
  None
  
  >>> person.age = 19
  >>> person.name = 'Scott_'
  >>> person.validate()
  *** ValidationError: Value "Scott_" did not match pattern "^[A-Za-z]+$".
  
  >>> person.name = 'Scott'
  >>> person.validate()
  None
  
  >>> person.nickname = None
  >>> person.validate()
  *** ValidationError: Field is required!
  
  >>> person.to_json_schema()
  {
      "additionalProperties": false,
      "properties": {
          "age": {
              "maximum": 101,
              "minimum": 18,
              "type": ["number", "null"]
          },
          "name": {
              "maxLength": 25,
              "minLength": 3,
              "pattern": "/^[A-Za-z]+$/",
              "type": "string"
          },
          "nickname": {,
              "type": ["string", "null"]
          }
      },
      "required": [
          "nickname",
          "name"
      ],
      "type": "object"
  }

  You can also validate scalars, when needed:

  >>> class Person(models.Base):
  ...
  ...     name = fields.StringField(
  ...         required=True,
  ...         validators=[
  ...             validators.Regex('^[A-Za-z]+$'),
  ...             validators.Length(3, 25),
  ...         ],
  ...     )
  ...     age = fields.IntField(
  ...         nullable=True,
  ...         validators=[
  ...             validators.Min(18),
  ...             validators.Max(101),
  ...         ]
  ...     )
  ...     nickname = fields.StringField(
  ...         required=True,
  ...         nullable=True
  ...     )
  ...
  
  >>> def only_odd_numbers(item):
  ... if item % 2 != 1:
  ...    raise validators.ValidationError("Only odd numbers are accepted")
  ...
  >>> class Person(models.Base):
  ... lucky_numbers = fields.ListField(int, item_validators=[only_odd_numbers])
  ... item_validator_str = fields.ListField(
  ...        str,
  ...        item_validators=[validators.Length(10, 20), validators.Regex(r"\w+")],
  ...        validators=[validators.Length(1, 2)],
  ...    )
  ...
  >>> Person.to_json_schema()
  {
      "type": "object",
      "additionalProperties": false,
      "properties": {
          "item_validator_str": {
              "type": "array",
              "items": {
                  "type": "string",
                  "minLength": 10,
                  "maxLength": 20,
                  "pattern": "/\\w+/"
              },
              "minItems": 1,
              "maxItems": 2
          },
          "lucky_numbers": {
              "type": "array",
              "items": {
                  "type": "number"
              }
          }
      }
  }

(Note that only_odd_numbers did not modify schema, since only class based validators are able to do that, though it will still work as expected in python. Use class based validators that can be expressed in json schema if you want to be 100% correct on schema side.)

• Lazy loading, best for circular references:

  >>> class Primary(models.Base):
  ...
  ...     name = fields.StringField()
  ...     secondary = fields.EmbeddedField('Secondary')
  
  >>> class Secondary(models.Base):
  ...
  ...    data = fields.IntField()
  ...    first = fields.EmbeddedField('Primary')

  You can use either Model, full path path.to.Model or relative imports .Model or ...Model.

• Using definitions to generate schema for circular references:

  >>> class File(models.Base):
  ...
  ...     name = fields.StringField()
  ...     size = fields.FloatField()
  
  >>> class Directory(models.Base):
  ...
  ...     name = fields.StringField()
  ...     children = fields.ListField(['Directory', File])
  
  >>> class Filesystem(models.Base):
  ...
  ...     name = fields.StringField()
  ...     children = fields.ListField([Directory, File])
  
  >>> Filesystem.to_json_schema()
  {
      "type": "object",
      "properties": {
          "name": {"type": "string"}
          "children": {
              "items": {
                  "oneOf": [
                      "#/definitions/directory",
                      "#/definitions/file"
                  ]
              },
              "type": "array"
          }
      },
      "additionalProperties": false,
      "definitions": {
          "directory": {
              "additionalProperties": false,
              "properties": {
                  "children": {
                      "items": {
                          "oneOf": [
                              "#/definitions/directory",
                              "#/definitions/file"
                          ]
                      },
                      "type": "array"
                  },
                  "name": {"type": "string"}
              },
              "type": "object"
          },
          "file": {
              "additionalProperties": false,
              "properties": {
                  "name": {"type": "string"},
                  "size": {"type": "number"}
              },
              "type": "object"
          }
      }
  }

• Dealing with schemaless data

(Plese note that using schemaless fields can cause your models to get out of control - especially if you are the one responsible for data schema. On the other hand there is usually the case when incomming data are with no schema defined and schemaless fields are the way to go.)

>>> class Event(models.Base):
...
...     name = fields.StringField()
...     size = fields.FloatField()
...     extra = fields.DictField()

>>> Event.to_json_schema()
{
    "type": "object",
    "additionalProperties": false,
    "properties": {
        "extra": {
            "type": "object"
        },
        "name": {
            "type": "string"
        },
        "size": {
            "type": "float"
        }
    }
}

DictField allow to pass any dict of values ("type": "object"), but note, that it will not make any validation on values except for the dict type.

• Compare JSON schemas:

  >>> from jsonmodels.utils import compare_schemas
  >>> schema1 = {'type': 'object'}
  >>> schema2 = {'type': 'array'}
  >>> compare_schemas(schema1, schema1)
  True
  >>> compare_schemas(schema1, schema2)
  False

More

For more examples and better description see full documentation: http://jsonmodels.rtfd.org.

jsonmodels's Issues

Use descriptors in field classes

prerequisites:
Now the model can be put invalid data. And we find out about it only after a full validation of the model. And the model would essentially be broken.

solution:
Use data descriptor and prevent placement in the model invalid data.

Remove data transformers

Values population

Problem with populating of values to EmbeddedField in constructor.

Embedded field

Embedded Field should allow to not give allowed types, so it can be used later more i.e. in schema generation.

Add `BoolField`.

_reset argument in populate method

populate() method should allow to reset all other fields to default values (_reset parameter maybe?)

Update contribute.rst.

Add EnumField

Add six to requirements

Right now after installing jsonmodels from pypi it does not install six what causes an error when you try to use it

Simple support for JSON-LD linked data fields

Is it feasible to add a simple level of compatibility for JSON-LD linked data fields, without diving into the JSON-LD spec (i.e. just treat the fields as standard JSON fields, but with some recognition of their special names and formats).

I'm thinking maybe a "LinkedDataField" that derives from BaseField that starts with the standard var name in Python, and then appends/transforms the JSON-LD decorators as appropriate:

'id' = '<some_URL>' in Python that's a LinkedDataField becomes '@id' in the JSON serialized form?

or 'ld_at_id' in Python becomes '@id'?

This might be very convenient, but shouldn't necessarily mistaken for "full support for JSON-LD"...

Does this seem like a good idea? Or would you rather "support JSON-LD 'the right way' or not at all"?

I could probably see my way to adding a pull request for adding simple field name transformation, with maybe a little guidance, but only if you think it's a reasonable approach to take.

Add `DateField`, `TimeField` and `DateTimeField`.

Allow validators to influence generated schema.

Add numeric field

Generation of JSON schema.

Generation of JSON schema should be possible on class, not on instance. Also during generation no instance should be created just to produce result.

Add predefined validators

• MinLength
• MaxLength
• Regex

Add full docs.

Use `anyjson` in project.

anyjson (https://pypi.python.org/pypi/anyjson) chooses fastest available json implementation - it would be usefull to use it instead of standard json library.

Check docs.

During test build of docs should be done. Also docs building should have spell checking enabled.

Add `EvaluateField`.

Add field that that's value is an evaluation of some function.

Probably this feature would need to rewrite whole structure to descriptors.

Add test for complexity.

Add test to ensure low cyclomatic complexity.
Fix wrong code to lower too high complexity (if needed).

Add help_text.

It would be easier for developers if you could specify help_text for fields.

Change test suite to PyTest

unittest is nice, yet it miss some fine features, really usefull during debugging.

Required property.

Think through required property in embedded field and list field - how to validate and put it into schema.

Fix validation of list field.

Ensure PEP8 compatibility.

Also add test to ensure it in future.

Add docs for validators.

And examples, also examples in schema.

Add LengthValidator

Validator should take two optionals arguments: min, max.

Add MaxValidator

It should take max and exclusive arguments.

Add the ability to import json schema

It would be great to be able to import the existing models in the format json scheme

Ensure PEP257 compatibility.

Also add test to ensure it in future.

Circular reference.

to_json_schema for circular references will fail.

Add MinValidator

It should take min and exclusive arguments.

Does 'required' list field should be required in JSONSchema?

Change fields to be descriptors.

Thanks to that change new features will be available. Also validation will be eager and some changes in inner API will be needed.

Types of fields.

Attribuge _types of fields should be rewritten to types but with backward compatibility.

Validation

Allow other types of validation (for now only required is checked).

Use six.with_metaclass in base

Rename error module

error module should be named errors.

BaseField children as ListField items_types

I try run simple code:

from jsonschema import models, fileds
>>> class TestModel(models.Base):
...     test_list_field = fields.ListField(fields.StringField())
... 
>>> test_obj = TestModel()
>>> test_obj.test_list_field = ['one', 'two', 3]
>>> test_obj.to_struct()
{'test_list_field': ['one', 'two', 3]}  # Validate pass (but should not)
>>> test_obj.to_json_schema()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hgen/.virtualenvs/igrofy-spec/lib/python3.4/site-packages/jsonmodels/models.py", line 73, in to_json_schema
    return parsers.to_json_schema(cls)
  File "/home/hgen/.virtualenvs/igrofy-spec/lib/python3.4/site-packages/jsonmodels/parsers.py", line 98, in to_json_schema
    prop[name] = _parse_list(field)
  File "/home/hgen/.virtualenvs/igrofy-spec/lib/python3.4/site-packages/jsonmodels/parsers.py", line 63, in _parse_list
    items = cls.to_json_schema()
AttributeError: 'StringField' object has no attribute 'to_json_schema'

Compare schemas.

Make compare_schemas public in main module.

Split requirements.

Split requirements into requirements.txt and requirements-dev.txt.

Also maybe do some file for whole pip freeze and make tests to ensure all deps are included.

Update documentation.

Change way of generation of jsonschema and similar

Probably also add it to generated schema.

Rename `error` module.

Module name should be consistent, so errors is better name.

Add json schema generation.

Model should be able to generate proper jsonschema for itself for all (at least) basic cases.

Rename method of fields

get_value_replacement -> get_default_value

Add default param for fields.

Probably also add it to generated schema.

Fix message when not allowed type is used.

While validation, if value has wrong type, ValidationError is raised, but error message is totally useless.

Add RegexValidator

Validator should take one argument: pattern

Refactor RegexValidator

Especially way of dealing with flags.