plynth / springfield

SpringField makes API data easy

SpringField

SpringField makes API data easy.

SpringField makes it simple to model structured data. Once the data is modeled, SpringField can parse API responses into easy to use Python objects and types. It can also generate the same structured data for making API request.

SpringField is ideal for:

• Restful JSON API data structures
• Parsing CSV data structures from csv.DictReader
• Turning anything Python can parse into a dict or list into a structured object

There is also a helper library for using SpringField with Mongo: springfield- mongo

Quickstart

To define an springfield.Entity, subclass springfield.Entity. Define your attributes by specifying fields. This library provides the follow self-describing fields to start with:

• IntField
• FloatField
• BooleanField
• StringField
• BytesField
• DateTimeField
• EmailField
• UrlField
• EntityField
• CollectionField

A quick example:

#!/usr/bin/env python
from springfield import Entity, fields
from springfield.timeutil import utcnow


class Bookmark(Entity):
    uri = fields.UrlField(doc='The bookmark uri.')
    verified = fields.BooleanField(doc='Whether or not this bookmark URI has been verified to exist.')
    added = fields.DateTimeField()


class User(Entity):
    id = fields.IntField(doc='Auto-incremented database id.')
    email = fields.EmailField(doc='The user\'s email address.')
    bookmarks = fields.CollectionField(fields.EntityField(Bookmark))
    created = fields.DateTimeField()


if __name__ == '__main__':
    user = User()
    user.id = 5
    user.email = '[email protected]'
    user.bookmarks = [
        {'uri': 'https://github.com'},
        {'uri': 'ftp://google.com', 'verified': True}
    ]
    user.created = utcnow()
    data = user.to_json()
    # `data` is suitable to return in something like a JSON API.
    print data

    # Similarly, `data` can be adapted from a JSON API request body.
    user = User.from_json(data)
    print user.email
    print user.created
    print user.bookmarks

Will print (the json was prettified to protect the innocent):

{
    "bookmarks":[
        {
            "uri":"https://github.com"
        },
        {
            "uri":"ftp://google.com",
            "verified":true
        }
    ],
    "created":"2017-01-25T20:25:54Z",
    "email":"[email protected]",
    "id":5
}
[email protected]
2017-01-25 20:47:37+00:00
[<Bookmark {uri: https://github.com}>, <Bookmark {verified: True, uri: ftp://google.com}>]

Notice a few things:

• Not every field is required for an entity. This is useful for doing sparse updates on an API.
• SpringField will adapt types in a non-destructive way.
• You can also create entities by adapting JSON, which is really handy at API boundaries.

Field Validation

SpringField does field validation when constructing entities, according to the types defined by the fields on that entity. For example:

You can define more complex field adaptation behavior by subclassing springfield.fields.Field and implementing your own fields. See the documentation on springfield.fields.Field for more information.

Similar Projects

• schematics (formerly dictshield)
• attrs

Changelog

0.9.1

• Fixed Entity.get() for Python 3

0.9.0

• Switched from future to six Python 2/3 compatibility libraries because future's modified str does not play well with adapters.

0.8.0

• Added support for Python 3.6+
• Dropped support for Python <2.7

0.7.17

• Fix packages for pytest plugin

0.7.16

• Allow EntityFields to use dotted-name class strings. This was done to allow circular references in entities that may refer to one another.
• Added BytesField

0.7.15

• Allow empty values for URL

springfield's Issues

Setting `Empty` on an Entity field fails

Setting Empty is allowed, but set tries to return the value that no longer exists.

    def set(self, instance, name, value):
        if value is Empty:
            if name in instance.__values__:
                del instance.__values__[name]
        else:
            instance.__values__[name] = self.adapt(value)
>       return instance.__values__[name]

UrlField with lowercase username and password

UrlField will lowercase the hostname in a URL. Which is fine. But because python urlparse includes the username, password in netloc the password gets lowercased too.

springfield does not play nice with pytest plugins

when trying to use springfield in a project, the pytest plugins are not getting picked up and pytest < 3.x silently fails to import it which causes other plugins to fail to import.

Default field values are not included in jsonify() output

If you create a field with a default value and call jsonify() on it without setting the field, the default value is not included in the result. Instead, the field is omitted.

Allow entity field composition without having to reference .field

Entities can reference fields of other entities:

class ExampleEntity(Entity):
    something = StringField()
    something_else = StringField()


class AnotherExampleEntity(Entity):
    something = ExampleEntity.something.field

It would be more obvious how to do this if you could do:

class AnotherExampleEntity(Entity):
    something = ExampleEntity.something

BytesField / UnicodeField ?

Might want to add field types that specifically use bytes or unicode. Or if its more forward-looking to python3, it would just be StringField (unicode) and BytesField.

`FlexEntity.flatten()` will convert unknown types to `str`

It should unknown types intact. Converting to str could be an encoding issue for unicode-like values. Values that are already Python types such as UUID should stay a UUID.

Instead, fields.get_field_for_type(val) should be used instead to guess the best type for the value.

https://github.com/six8/springfield/blame/develop/src/springfield/entity.py#L356

        elif not isinstance(val, (basestring, int, float, long)):
            val = str(val)

Make a public method to add type maps for custom fields

If you add a custom field, you have to do this so FlexEntity.jsonify will handle it:

from springfield.fields import _type_map
import UuidField

_type_map[UUID] = UuidField()

A public method like fields.add_type_map(UUID, UuidField()) would be better.

Need to implement __hash__ so entities work with sets

Can't use a FlexEntity as the "kwargs" part of a call

e.g.

>>> (lambda **args: args)(**FlexEntity(foo=1, bar=2))
Traceback (most recent call last):
  File "<input>", line 1, in <module>
TypeError: attribute of type 'NoneType' is not callable

The solution is to implement the keys() method:

>>> FlexEntity.keys = (lambda e: [key for key, value in e.iteritems()])
>>> (lambda **args: args)(**FlexEntity(foo=1, bar=2))
{'foo': 1, 'bar': 2}

When field has a date with tz and you try to update without a timezone, springfield should catch and treat as a changed value instead of raising an error

  File "/cellsec/virtualenv/lib/python2.7/site-packages/springfield/fields.py", line 43, in __set__
    if new_value != old_value:
TypeError: can't compare offset-naive and offset-aware datetimes

This error should be caught and treated as not-equal.

Can't use @propertys on Entities

Python's __setattr__ has precedence over descriptors like @property. If you do, you'll get:

  File "/python2.7/site-packages/springfield/entity.py", line 203, in __setattr__
    raise AttributeError('Field %r not defined.' % name)

Entity needs to be modified to allow properties:

if name in self.__fields__ or isinstance(getattr(self.__class__, name, None), property):
    object.__setattr__(self, name, value)
else:
    raise AttributeError('Field %r not defined.' % name)

As a workaround, the following can be added to any Entity to support properties:

def __setattr__(self, name, value):
        if name not in self.__fields__ and isinstance(getattr(self.__class__, name, None), property):
            object.__setattr__(self, name, value)
        else:
            super(EntityName, self).__setattr__(name, value)

Entities do not play nicely with @property setters: AttributeError

from springfield import Entity, fields

class TestEntity(Entity):
    foo = fields.StringField()

    @property
    def bar(self):
        return True

    @bar.setter
    def bar(self, value):
        print 'Set bar to %s' % value

e = TestEntity()
print e.bar
e.bar = False

True
Traceback (most recent call last):
  File "/Users/mthornton/Library/Preferences/PyCharm2017.3/scratches/scratch.py", line 16, in <module>
    e.bar = False
  File "/Users/mthornton/Projects/PulseSecure/pulse-one-pycharm/venv/lib/python2.7/site-packages/springfield/entity.py", line 203, in __setattr__
    raise AttributeError('Field %r not defined.' % name)
AttributeError: Field 'bar' not defined.

FlexEntity does not support __deepcopy__

>>> from springfield import FlexEntity
>>> deepcopy(FlexEntity())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/copy.py", line 182, in deepcopy
    rv = reductor(2)

See also https://stackoverflow.com/questions/49901590/python-using-copy-deepcopy-on-dotdict/49902096#49902096

ThirdPartyInfoEntity should use DateTimeField for its release_date

Currently ThirdPartyInfoEntity used StringField for its release_date whereas it should use DateTimeField.

This causes the following issue:

• release_date (2018-07-03): Could not adapt datetime.date(2018, 7, 3) to <type 'unicode'>

Alias does not seem to work

I tried to use alias but it didn't seem to work. I wrote some test cases and they all fail. See:

dobesv@2df49ab

For the tests. The alias never operates as expected. Maybe I'm not using it properly?