klen / marshmallow-peewee

Peewee ORM integration with the marshmallow (de)serialization library.

Marshmallow-Peewee

Marshmallow-Peewee -- Peewee ORM integration with the Marshmallow (de)serialization library.

Requirements

• python >= 3.7

Installation

marshmallow-peewee should be installed using pip:

$ pip install marshmallow-peewee

Quickstart

    import peewee as pw


    class Role(pw.Model):
        name = pw.CharField(255, default='user')


    class User(pw.Model):

        created = pw.DateTimeField(default=dt.datetime.now())
        name = pw.CharField(255)
        title = pw.CharField(127, null=True)
        active = pw.BooleanField(default=True)
        rating = pw.IntegerField(default=0)

        role = pw.ForeignKeyField(Role)


    from marshmallow_peewee import ModelSchema

    class UserSchema(ModelSchema):

        class Meta:
            model = User

    role = Role.create()
    user = User.create(name='Mike', role=role)

    result = UserSchema().dump(user)
    print(result)
    # {'active': True,
    #  'created': '2016-03-29T15:27:18.600034+00:00',
    #  'id': 1,
    #  'name': 'Mike',
    #  'rating': 0,
    #  'role': 1,
    #  'title': None}

    result = UserSchema().load(result)
    assert isinstance(result, User)
    assert result.name == 'Mike'

    from marshmallow_peewee import Related

    class UserSchema(ModelSchema):

        role = Related()

        class Meta:
            model = User

    result = UserSchema().dump(user)
    print(result)
    # {'active': True,
    #  'created': '2016-03-29T15:30:32.767483+00:00',
    #  'id': 1,
    #  'name': 'Mike',
    #  'rating': 0,
    #  'role': {'id': 5, 'name': 'user'},
    #  'title': None}

    result = UserSchema().load(result)
    assert isinstance(result, User)
    assert isinstance(result.role, Role)

Usage

    import peewee as pw


    class Role(pw.Model):
        name = pw.CharField(255, default='user')


    class User(pw.Model):

        created = pw.DateTimeField(default=dt.datetime.now())
        name = pw.CharField(255)
        title = pw.CharField(127, null=True)
        active = pw.BooleanField(default=True)
        rating = pw.IntegerField(default=0)

        role = pw.ForeignKeyField(Role)


    from marshmallow_peewee import ModelSchema

    class UserSchema(ModelSchema):

        class Meta:

            # model: Bind peewee.Model to the Schema
            model = User

            # model_converter: Use custom model_converter
            # model_converter = marshmallow_peewee.DefaultConverter

            # dump_only_pk: Primary key is dump only
            # dump_only_pk = True

            # string_keys: Convert keys to strings
            # string_keys = True

            # id_keys: serialize (and deserialize) ForeignKey fields with _id suffix
            # id_keys = False

You may set global options for marshmallow-peewee:

from marshmallow_peewee import setup

setup(id_keys=True, string_keys=False)  # Set options for all schemas

class UserSchema(ModelSchema):
  # ...

Customize fields convertion:

from marshmallow_peewee import DefaultConverter

# Customize global

# Serialize boolean as string
DefaultConverter.register(peewee.BooleanField, marshmallow.fields.String)

# Alternative method
@DefaultConverter.register(peewee.BooleanField)
def build_field(field: peewee.Field, opts, **field_params):
  return marshmallow.fields.String(**params)

# Customize only for a scheme

class CustomConverter(DefaultConverter):
  pass


CustomConverter.register(...)


class CustomSchema(ModelSchema): # may be inherited
  class Meta:
    model_converter = CustomConverter

Bug tracker

If you have any suggestions, bug reports or annoyances please report them to the issue tracker at https://github.com/klen/marshmallow-peewee/issues

Contributing

Development of the project happens at: https://github.com/klen/marshmallow-peewee

License

Licensed under a MIT License

marshmallow-peewee's Issues

ForeignKey field with `_id` suffix

Hi. I think marshmallow-peewee may provide option that serialize (and deserialize) ForeignKey fields with _id suffix.

On API we have:

{
  "id": "1",
  "title": "The Chronicles of Narnia",
  "author": "1"
}

For me is not clear if "1" is author pseudonym or id of another model named author.

{
  "id": 1,
  "title": "The Chronicles of Narnia",
  "author_id": 1
}

I'll be trying to hack on my own, but I'm not sure if you want this feature as part of package.

Related field seriaizes as an integer even if the primary key is a string

From fields.py

    def _deserialize(self, value, attr, data, **kwargs):
        if isinstance(value, str):
            return int(value)

        return super(Related, self)._deserialize(value, attr, data)

But in my models.py, my primary key is a StringField, so this breaks.

A fix would be to either look at the field type of the other ModelSchema it's "Related" to, or at least let the user pass in a parameter to serialize it to.

Serializer with a Related() field only goes one inheritance, but does not cascade further

I have more that one model schema, each has a Related() field, which link to another, and so on. Once the object is dumped via a serializer, it only cascades to one related model via the Foreign key, but not to the next model.

Here is an example of the code to replicate the issue:

import peewee as pw
from marshmallow_peewee import ModelSchema, Related

db = pw.SqliteDatabase('example.db')

# Define models
class BaseModel(pw.Model):
    class Meta:
        database = db

class Planet(BaseModel):
    name = pw.CharField()

class Country(BaseModel):
    planet = pw.ForeignKeyField(Planet, backref='countries')
    name = pw.CharField()

class City(BaseModel):
    country = pw.ForeignKeyField(Country, backref='cities')
    name = pw.CharField()

db.create_tables([Planet, Country, City])

# Populate some data
planet = Planet(name='Earth')
planet.save()

country = Country(name='Wonderland', planet=planet)
country.save()

city = City(name='Springfield', country=country)
city.save()

# Create serializers
class PlanetSchema(ModelSchema):
    class Meta:
        model = Planet
    countries = Related()

class CountrySchema(ModelSchema):
    class Meta:
        model = Country
    cities = Related()

class CitySchema(ModelSchema):
    class Meta:
        model = City

# Now dump data using serializer
planet = Planet.get(1)
print(PlanetSchema().dump(planet)[0])

The output from the code above is:

{
    'id': '1', 
    'name': 'Earth',
    'countries': [
        {
            'id': '1', 
            'name': 'Wonderland', 
            'planet': '1'
        }
    ]
}

However, the desired output is this one:

{
    'id': '1', 
    'name': 'Earth',
    'countries': [
        {
            'id': '1', 
            'name': 'Wonderland', 
            'planet': '1',
            'cities': [
                {'id': '1', 'country': '1', 'name': 'Springfield'}
            ]
        }
    ]
}

As you may see, the desired outcome begins from Planet, cascades to Country, which later cascades to City.

If the CountrySchema serializer is used to dump it, it then includes the CitySchema as well, but not when the PlanetSchema is used.

marshmallow.exceptions.ValidationError: {'id': ['Unknown field.']}

The Output/Traceback

1
{'active': True,
 'created': '2022-06-02T21:04:07.061651',
 'id': '1',
 'name': 'Mike',
 'rating': 0,
 'role': '1',
 'title': None}
Traceback (most recent call last):
  File "/Users/cforadas/PycharmProjects/flask_blueprint_testing/app.py", line 49, in <module>
    result = UserSchema().load(result)
  File "/Users/cforadas/PycharmProjects/flask_blueprint_testing/venv/lib/python3.10/site-packages/marshmallow_peewee/schema.py", line 90, in load
    return super(ModelSchema, self).load(data, *args, **kwargs)
  File "/Users/cforadas/PycharmProjects/flask_blueprint_testing/venv/lib/python3.10/site-packages/marshmallow/schema.py", line 722, in load
    return self._do_load(
  File "/Users/cforadas/PycharmProjects/flask_blueprint_testing/venv/lib/python3.10/site-packages/marshmallow/schema.py", line 909, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'id': ['Unknown field.']}

Process finished with exit code 1

Python Version

'3.10.4 (v3.10.4:9d38120e33, Mar 23 2022, 17:29:05) [Clang 13.0.0 (clang-1300.0.29.30)]'

Library Versions

marshmallow==3.16.0
Marshmallow-Peewee==3.2.1
peewee==3.14.10

Code

1. import pprint
2. 
3. import peewee as pw
4. from peewee import SqliteDatabase
5. from marshmallow_peewee import ModelSchema
6. from marshmallow_peewee import Related
7. import datetime as dt
8. 
9. db = SqliteDatabase(':memory:')
10. 
11. 
12. class Role(pw.Model):
13.     name = pw.CharField(255, default='user')
14. 
15.     class Meta:
16.         database = db # This model uses the "people.db" database.
17. 
18. 
19. class User(pw.Model):
20.     created = pw.DateTimeField(default=dt.datetime.now())
21.     name = pw.CharField(255)
22.     title = pw.CharField(127, null=True)
23.     active = pw.BooleanField(default=True)
24.     rating = pw.IntegerField(default=0)
25.     role = pw.ForeignKeyField(Role)
26. 
27.     class Meta:
28.         database = db # This model uses the "people.db" database.
29. 
30. 
31. class UserSchema(ModelSchema):
32.     class Meta:
33.         model = User
34. 
35. 
36. # create the tables
37. db.create_tables([Role, User])
38. 
39. # create role
40. role = Role.create()
41. print(role)
42. 
43. # create a user
44. user = User.create(name='Mike', role=role.id)
45. 
46. result = UserSchema().dump(user)
47. pprint.pprint(result)
48. 
49. result = UserSchema().load(result)
50. assert isinstance(result, User)
51. assert result.name == 'Mike'
52. 
53. 
54. class UserSchema(ModelSchema):
55.     role = Related()
56. 
57.     class Meta:
58.         model = User
59. 
60. 
61. result = UserSchema().dump(user)
62. print(result)
63. 
64. result = UserSchema().load(result)
65. assert isinstance(result, User)
66. assert isinstance(result.role, Role)
67. 

Comments

I happend to notice that the results 'id' is a string and not an integer unlike in your example code (quickstart).

object has no attribute '_data'

I'm getting this error when trying to serialize queryset with schema that doesn't define fields:

File "...\site-packages\marshmallow_peewee\fields.py", line 79, in get_value
value = obj._data.get(attr)
AttributeError: 'Country' object has no attribute '_data'

and was able to fix it by changing it to:
value = obj.data.get(attr)

OS: win7 pro
Python 3.6.3
peewee:3.2.5
marshmallow: 2.15.0
Marshmallow-Peewee: 1.2.7

[idea] Filter for non required fields with empty incoming data

Hi!
Sometimes we've got an empty field value in incoming data. And if this field maps on database model field which has default value or null param is set to True, empty value is not good for us. Not always, but very often. I use this code in my ModelSchema wrapper:

class ModelSchema(_ModelSchema):

    @pre_load
    def drop_empty_fields(self, item):
        field_names = tuple(item.keys())
        for k in field_names:
            field = self.fields.get(k)
            if field and not field.required and not item[k]:
                item.pop(k)

For example: field is DateField(default=datetime.date.today), incoming data value is "" (empty string). Would be nice to drop this field from incoming dataset and save the instance with default value. Now we get an error Not a valid date.
I suggest to add a some boolean initial param (for ex. filter) for manage this case.
What do you think?

[question] String type for AutoField

What the point of using marshmallow String type for peewee AutoField instead of Integer?
Is it bug or feature?

marshmallow-peewee/marshmallow_peewee/convert.py

Line 15 in 595867d

(pw.AutoField, fields.String),

Marshmallow 3.0.0rc3 incompatibility

Looks like marshmallow changed out from under marshmallow-peewee:

ImportError while loading conftest 'tests/conftest.py'.
[...]
.venv/lib/python3.7/site-packages/marshmallow_peewee/__init__.py:9: in <module>
    from .schema import ModelSchema # noqa
.venv/lib/python3.7/site-packages/marshmallow_peewee/schema.py:5: in <module>
    from .convert import ModelConverter
.venv/lib/python3.7/site-packages/marshmallow_peewee/convert.py:7: in <module>
    from .fields import ForeignKey
.venv/lib/python3.7/site-packages/marshmallow_peewee/fields.py:5: in <module>
    from marshmallow.compat import PY2, string_types
E   ImportError: cannot import name 'string_types' from 'marshmallow.compat' (.venv/lib/python3.7/site-packages/marshmallow/compat.py)

marshmallow-code/marshmallow@d9bcc48 is the change.

"only" modifier for schema is ignored

In marshmallow, there's a quick way to filter which fields to show in the dump, via "only" option (http://marshmallow.readthedocs.io/en/latest/quickstart.html#filtering-output).

But this is ignored in the ModelSchema, even when the base options are being parsed and it should be recognized and used.

.make_instance() got an unexpected keyword argument 'many'

Marshmallow from version 3.0.0 when decorators such as validate_schema is used, always pass many and partial arguments. Can you add or pass them in ModelSchema.make_instance ?

Object from Schema not full

Schema with custom full_path and url

class FileSchema(ModelSchema):

    storage = Related(required=True)
    mime = Related()
    user_id = fields.Int()
    user = Related(load_only=True)
    full_path = fields.FormattedString('{path}{name}')
    url = fields.FormattedString(S3_ENDPOINT_URL +'/{storage.bucket}/{path}{name}')

    class Meta:
        model = File

but, when load object

        fileSchema = FileSchema()
        file, errors = fileSchema.load(payload['file'])
        print(json.dumps(file.__dict__, indent=4, sort_keys=True, default=str))

i don't have file_path, only base File model fields

{
    "__data__": {
        "created_at": "2018-06-28 16:37:19",
        "hash": "Y9cR-xxxxxxxxx",
        "name": "vid20180619103313.mp4",
        "path": "xxxxxxxxxxxxx/",
        "size": null,
        "storage": null,
        "updated_at": "2018-06-28 16:37:19",
        "user": xxxxxx
    },
    "__rel__": {
        "storage": "None"
    },
    "_dirty": "{'user', 'created_at', 'path', 'name', 'storage', 'hash', 'size', 'updated_at'}",
    "url": "https://xxxxxxxxxxxxxx/vid20180619103313.mp4"
}

if dump

        result, errors = fileSchema.dump(file)
        print(type(result))
        print(json.dumps(result, indent=4, sort_keys=True))

all ok

{
    "created_at": "2018-06-28 16:37:19",
    "hash": "Y9cR-xxxxxxxxxxxx",
    "id": 7200,
    "mime_id": null,
    "name": "vid20180619103313.mp4",
    "path": "xxxxxxxxxxxxxxxx/",
    "size": null,
    "storage": {
        "bucket": "files",
        "id": 13,
        "type": "s3",
        "url": "xxxxxxxxxx/files"
    },
    "storage_id": 13,
    "updated_at": "2018-06-28 16:37:19",
    "url": "https://xxxxxxxxxxxxxxxxxxx/vid20180619103313.mp4",
    "user_id": xxxxxx
}

why not in object?

[question] ForeignKey converter

Would it be correct to use the type of field it refers to for a foreign key field?

class ModelConverter(object):
    ...
    def convert_ForeignKeyField(self, field, **params):
        return self.convert_default(field.rel_field, **params)

Support for Marshmallow version 3.0.0

Currently marshmallow-peewee supports marshmallow<=3.0.0rc5, but not higher release candidates.

As a consumer, I would like this library to support the latest release of Marshmallow once a stable version is published.
The latest release now is version 3.0.0rc9.

Here is a pending PR to address what was removed from Marshmallow, in order to support version 2 and 3:
#78

primary and foreign keys are strings

If I run the example from the readme I get the following json:

{
'active': True,
'rating': 0,
'role': '7',
'name': 'Mike',
'title': None,
'id': '7',
'created': '2018-12-13T17:37:30.956255+00:00'
}

In the readme the keys role and id are integers, but in fact they get converted to strings. It would be nice if they where integers.

ID field required in schema

For creating new instance from data ID field should not be mandatory.

[dell] ~/projects/PHD $ flask shell
Python 3.6.5 (default, Apr 12 2018, 22:45:43) 
[GCC 7.3.1 20180312] on linux
App: app.core [debug]
Instance: /home/ak04nv/projects/PHD/instance
>>> from app.samples.schema import SampleSchema
>>> obj, err = SampleSchema().load({})
>>> err
{'id': ['Missing data for required field.'], 'name': ['Missing data for required field.']}
>>> 

It was on 1.2.1 version