Coder Social home page Coder Social logo

sqlalchemy-jsonapi's Introduction

SQLAlchemy-JSONAPI

Build Status

JSON API implementation for use with SQLAlchemy.

SQLAlchemy-JSONAPI aims to implement the JSON API spec and to make it as simple to use and implement as possible.

Installation

pip install sqlalchemy-jsonapi

Quick usage with Flask-SQLAlchemy

# Assuming FlaskSQLAlchemy is db and your Flask app is app:
from sqlalchemy_jsonapi import FlaskJSONAPI

api = FlaskJSONAPI(app, db)

# Or, for factory-style applications
api = FlaskJSONAPI()
api.init_app(app, db)

Quick usage without Flask

# Assuming declarative base is called Base
from sqlalchemy_jsonapi import JSONAPI
api = JSONAPI(Base)

# And assuming a SQLAlchemy session
print(api.get_collection(session, {}, 'resource-type'))

sqlalchemy-jsonapi's People

Contributors

anderycks avatar angelosarto avatar bladams avatar coltonprovias avatar duk3luk3 avatar kaitj91 avatar rockstar avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

sqlalchemy-jsonapi's Issues

JSON API RC3 tracking

Hey there! I just wanted to let you know that JSON API has hit RC3, and we'd like client libraries to start implementing it like it was 1.0, as a final check that we're good for 1.0. I wanted to make this issue to let you know!

json-api/json-api#484

5.0.0 Changes

General Changes:

  • Addition of Ember-compatible filtering
  • Query modification
  • Extended filtering
  • Default pagination settings
  • Binary data serialization
  • Optional: MessagePack

Endpoint Changes:

  • GET Collection
    • Bad query parameter handling
    • Filtering
  • GET Resource
    • Related Inclusions
    • Sparse Fieldsets
    • Bad query parameter
  • GET Related
    • Sparse Fieldsets
    • Related Includes
    • Sorting
    • Pagination
    • Filtering
    • Bad query parameter
  • GET Relationship
    • Sorting
    • Pagination
    • Filtering
    • Bad query parameter
  • POST Collection
    • Bad query parameter
  • POST Relationship
    • Sorting
    • Pagination
    • Filtering
    • Bad query parameter
  • PATCH Resource
    • Sparse Fieldsets
    • Related Includes
    • Bad query parameter
  • PATCH Relationship
    • Sorting
    • Pagination
    • Filtering
    • Bad query parameter
  • DELETE Resource
    • Bad query parameter
  • DELETE Relationship
    • Sorting
    • Pagination
    • Filtering
    • Bad query parameter

cannot import name make_response

Hi,
I'm trying to test your package, but I have problem with import make_response method from flask, because python is trying import the module from /sqlalchemy-jsonapi/flask.py.

Code:

from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
from sqlalchemy_jsonapi import FlaskJSONAPI
app = Flask(__name__)
db = SQLAlchemy(app)
api = FlaskJSONAPI(app, db)
@app.route("/")
def hello():
     return "Hello World!"
if __name__ == "__main__":
    app.run(debug=True)

Error:

Traceback (most recent call last):
  File "D:/workspace/web_development/project/restfull-test/application.py", line 3, in <module>
    from sqlalchemy_jsonapi import FlaskJSONAPI
  File "C:\Python27\lib\site-packages\sqlalchemy_jsonapi\__init__.py", line 1, in <module>
    from .flask import FlaskJSONAPI
  File "C:\Python27\lib\site-packages\sqlalchemy_jsonapi\flask.py", line 14, in <module>
    from flask import make_response, request
ImportError: cannot import name make_response

Patch_relationship is not raising BadRequestError when payload data is not array of dictionaries

It is very important that the library is able to return some sort of response that let's the user know they submitted an invalid request by raising a BadRequestError and returning a 400 and some sort of description explaining why.

Through testing patch_relationship, I found that we are not raising a BadRequestError when we send a payload where data is not an array of dictionaries.

For example:

       def test_patch_relationship_with_invalid_data_format(self):
        """Patch relationship when data is not an array of dictionaries returns 400.

        A BadRequestError is raised.
        """
        user = models.User(
            first='Sally', last='Smith',
            password='password', username='SallySmith1')
        self.session.add(user)
        blog_post = models.Post(
            title='This Is A Title', content='This is the content',
            author_id=user.id, author=user)
        self.session.add(blog_post)
        comment = models.Comment(
            content='This is comment 1', author_id=user.id,
            post_id=blog_post.id, author=user, post=blog_post)
        self.session.add(comment)
        self.session.commit()
        payload = {
            'data': ['foo']
        }

        with self.assertRaises(errors.BadRequestError) as error:
            models.serializer.patch_relationship(
                self.session, payload, 'posts', blog_post.id, 'comments')

        self.assertEqual(error.exception.detail, 'data must be an array of objects')
        self.assertEqual(error.exception.status_code, 400)

This test fails because there is no check for this in place in patch_relationship. Thus when I try to run this test a TypeError occurs. Specifically the TypeError is occuring because we are failing to check

if not isinstance(item, dict):
    raise BadRequestError('data must be an array of objects')

at the beginning of the for loop below:

        for item in json_data['data']:
            to_relate = self._fetch_resource(
                session, item['type'], item['id'], Permissions.EDIT)

Because we do not have this check a TypeError occurs because we are trying to look up item['type'] and because it is a list we would need to look up by integer indices and not strings.
This is an easy fix that would provide an informative error message to users who send badly formatted payloads.

Unclear error message on post_collection with OneToMany relationship

For background information let's look at this test:

    def test_add_resource_with_invalid_relationships(self):
        """Create resource with invalid relationship returns 400.

        A BadRequestError is raised.
        """
        payload = {
            'data': {
                'attributes': {
                    'first': 'Sally',
                    'last': 'Smith',
                    'username': 'SallySmith1',
                    'password': 'password',
                },
                'type': 'users',
                'relationships': {
                    'posts': {
                        'data': {
                            'type': 'posts',
                            'id': 1
                        }
                    }
                }
            }
        }

        with self.assertRaises(errors.BadRequestError) as error:
            models.serializer.post_collection(
                self.session, payload, 'users')

        self.assertEqual(error.exception.detail, 'posts must be an array')
        self.assertEqual(error.exception.status_code, 400)

As we can see the error message produced by the library is somewhat confusing.
If we actually look at the check that is occuring it is:

                    if not isinstance(data_rel, list):
                        raise BadRequestError(
                            '{} must be an array'.format(key)

This is interesting because if we actually look at what data_rel is, it is defined as we see by this chunk of code:

                data_rel = data['data']['relationships'][api_key]
                if 'data' not in data_rel.keys():
                    raise BadRequestError(
                        'Missing data key in relationship {}'.format(key))
                data_rel = data_rel['data']

so data_rel is actually {'type': 'posts', 'id': 1}. Thus when the check to see if data_rel is a list, it is not but the detail of the BadRequestError is saying that the 'posts must be an array', since 'posts' is the key.

I believe that this detail message should be changed to say 'posts data must be an array'.
This might be trivial for someone that knows jsonapi well. However, if someone were trying to simply figure out how to format their response correctly after receiving this error message it would be confusing.

relationship_descriptor broken due to c&p error

Hello,

first of all: Thank you for this library, it's extremely useful for me. ๐Ÿ‘ โค๏ธ

The issue I found:

Using relationship_descriptor in sqlalchemy-jsonapi 4.0.8 causes this error that is also present in current master:

Traceback (most recent call last):
  File "/home/erlacher/git/rbg/lxcomputing/lxcp/__init__.py", line 19, in <module>
    api = FlaskJSONAPI(app, db)
  File "/home/erlacher/git/rbg/lxcomputing/.venv/lib/python3.6/site-packages/sqlalchemy_jsonapi/flaskext.py", line 103, in __init__
    self._setup_adapter(namespace, route_prefix)
  File "/home/erlacher/git/rbg/lxcomputing/.venv/lib/python3.6/site-packages/sqlalchemy_jsonapi/flaskext.py", line 166, in _setup_adapter
    self.serializer = JSONAPI(self.sqla.Model, prefix='{}://{}{}'.format(self.app.config['PREFERRED_URL_SCHEME'], self.app.config['SERVER_NAME'], route_prefix))
  File "/home/erlacher/git/rbg/lxcomputing/.venv/lib/python3.6/site-packages/sqlalchemy_jsonapi/serializer.py", line 262, in __init__
    rels_desc.setdefault(attribute, defaults)
UnboundLocalError: local variable 'attribute' referenced before assignment

I think this is caused by a copy-paste error on this line: https://github.com/ColtonProvias/sqlalchemy-jsonapi/blob/master/sqlalchemy_jsonapi/serializer.py#L269 - attribute needs to be relationship.

Reflection

Can I use reflection (from database instance) with this library? The example has specific models only. Thx!

Autoflush on collection post

I've had this issue where sqlalchemy session would get flushed in the middle of relationships setters. Since data attributes are not yet set at that point, the sql insert query fails, aborting the post request.
Disabling autoflush fixed this: emilecaron@17742de

I'm handling some fairly complex models, with a lot of validation and relations so this bug should not concern most users.

Post_relationship is not raising BadRequestError when payload data is not array of dictionaries

It is very important that the library is able to return some sort of response that let's the user know they submitted an invalid request by raising a BadRequestError and returning a 400 and some sort of description explaining why.

Through testing post_relationship, I found that we are not raising a BadRequestError when we send a payload where the data is not an array of dictionaries.

For example:

        def test_post_relationship_when_data_has_invalid_format(self):
        """Post relationship when data is not an array of dictionaries returns 400.

        A BadRequestError is raised.
        """
        user = models.User(
            first='Sally', last='Smith',
            password='password', username='SallySmith1')
        self.session.add(user)
        blog_post = models.Post(
            title='This Is A Title', content='This is the content',
            author_id=user.id, author=user)
        self.session.add(blog_post)
        comment = models.Comment(
            content='This is the first comment',
            author_id=user.id, author=user)
        self.session.add(comment)
        self.session.commit()
        payload = {
            'data': ['test']
        }

        with self.assertRaises(errors.BadRequestError) as error:
            models.serializer.post_relationship(
                self.session, payload, 'posts', blog_post.id, 'comments')

        self.assertEqual(
            error.exception.detail, 'data must be an array of objects')
        self.assertEqual(error.exception.status_code, 400)

This test fails because there is no check for this in place in post_relationship. Thus when I try to run this test a AttributeError occurs. Specifically the AttributeError is occuring because we are failing to check

if not isinstance(item, dict):
raise BadRequestError('data must be an array of objects')

at the beginning of the for loop below:

                for item in json_data['data']:
                    if {'type', 'id'} != set(item.keys()):
                        raise BadRequestError(
                            '{} must have type and id keys'.format(relationship.key))

Because we do not have this check a AttributeError occurs because the item does not have any keys when trying to execute item.keys().
This is an easy fix that would provide an informative error message to users who send badly formatted payloads.

Mismatched function argument types.

According to the documentation, a call to get_collection should look like:

serializer.get_collection(db.session, {}, 'users')

Where the string users is eventually passed to the inject_model decorator which looks like:

def wrapped(serializer, session, data, params):
    api_type = params.pop('api_type')

Currently this is located at: https://github.com/ColtonProvias/sqlalchemy-jsonapi/blob/master/sqlalchemy_jsonapi/serializer.py#L196

So params will be the string users in the example above. It looks like the expectation is that params should be a dict of keyword arguments, like:

def wrapped(serializer, session, data, **params):
    api_type = params.pop('api_type')

However, get_collection looks like:

def get_collection(self, session, query, model):

There is no api_type keyword argument. This assumption is made by other, similar functions too.

This:

def wrapped(serializer, session, data, api_type):

worked for me. It seems like api_type can't be optional (and therefore should be a positional argument).

I didn't see a coverage report in the Travis CI output. I think I see where this is covered in the tests. However I'm baffled by how this could be covered and pass.

FlaskJSONAPI is None/doesn't import properly?

Doesn't appear to import properly:

Traceback (most recent call last):
File "./manage.py", line 11, in
os.getenv('MY_APP_CONFIG', 'default'), app, database)
File "/Users/ricorodriguez/Desktop/git/project/my_app/webapp.py", line 25, in setup_app
return FlaskJSONAPI(app, database)
TypeError: 'NoneType' object is not callable

Here's the dir() output for the sqlalchemy_jsonapi module:

['ALL_PERMISSIONS', 'AttributeActions', 'Endpoint', 'FlaskJSONAPI', 'INTERACTIVE_PERMISSIONS', 'JSONAPI', 'Method', 'Permissions', 'RelationshipActions', 'builtins', 'doc', 'file', 'name', 'package', 'path', 'attr_descriptor', 'constants', 'errors', 'permission_test', 'relationship_descriptor', 'serializer']

Looks fine to me - what could be going on here?

Incompatability with sqlalchemy-i18n?

When I switch from 3.0.2 to any 4.X series I get a new error on startup:

sqlalchemy_i18n.exc.UnknownLocaleError: Unknown locale jsonapi_desc_for_attrs given for instance of class <class 'sqlalchemy.ext.declarative.api.DeclarativeMeta'>. Locale is not one of ['zxx', 'en_us', 'de_de']

When I downgrade back to 3.0.2 it works fine so there is something definitely changed in how this integrates in 4.X vs 3.X.

I don't yet have an isolated test case this is integrated into a whole bunch of code. I will work on creating one as soon as I can.

Do you have any places or configurations I should check until I get a better example?

Delete_relationship does not raise BadRequestError when data is not array of dictionaries

It is very important that the library is able to return some sort of response that let's the user know they submitted an invalid request by raising a BadRequestError and returning a 400 and some sort of description explaining why.

Through testing delete_relationship, I found that we are not raising a BadRequestError when we send a payload where data is not an array of dictionaries.

For example:

    def test_delete_one_to_many_relationship_when_data_has_invalid_format(self):
        """Delete a relationship when the payload data is not an array of dictionaries returns 400.

        A BadRequestError is raised.
        """
        user = models.User(
            first='Sally', last='Smith',
            password='password', username='SallySmith1')
        self.session.add(user)
        blog_post = models.Post(
            title='This Is A Title', content='This is the content',
            author_id=user.id, author=user)
        self.session.add(blog_post)
        comment = models.Comment(
            content='This is a comment', author_id=user.id,
            post_id=blog_post.id, author=user, post=blog_post)
        self.session.add(comment)
        self.session.commit()
        payload = {
            'data': ['test']
        }

        with self.assertRaises(errors.BadRequestError) as error:
            models.serializer.delete_relationship(
                self.session, payload, 'posts', blog_post.id, 'comments')

       
        self.assertEqual(error.exception.detail, 'data must be an array of objects')
        self.assertEqual(error.exception.status_code, 400)

This test fails because there is no check for this in place in delete_relationship. Thus when I try to run this test a TypeError occurs. Specifically the TypeError is occuring because we are failing to check

if not isinstance(item, dict):
    raise BadRequestError('data must be an array of objects')

at the beginning of the for loop below:

        for item in data['data']:
            item = self._fetch_resource(session, item['type'], item['id'],
                                        Permissions.EDIT)

Because we do not have this check a TypeError occurs because we are trying to look up item['type'] and because it is a list we would need to look up by integer indices and not strings.
This is an easy fix that would provide an informative error message to users who send badly formatted payloads.

Post_collection is not raising BadRequestError when payload relationships is not a dictionary

It is very important that the library is able to return some sort of response that let's the user know they submitted an invalid request by raising a BadRequestError and returning a 400 and some sort of description explaining why.

Through testing post_collection, I found that we are not raising a BadRequestError when we send a payload where the relationships is an array rather than a dictionary

For example:

    def test_add_resource_when_payload_relationships_is_array(self):
        """Create a resource when payload relationships is an array returns 400.

        The relationships must be of type dictionary.
        A BadRequestError is raised.
        """
        user = models.User(
            first='Sally', last='Smith',
            password='password', username='SallySmith1')
        self.session.add(user)
        self.session.commit()

        payload = {
            'data': {
                'type': 'posts',
                'attributes': {
                    'title': 'Some Title',
                    'content': 'Some Content Inside'
                },
                'relationships': [{
                    'author': {
                        'data': {
                            'type': 'users',
                            'id': 1,
                        }
                    }
                }]
            }
        }

        with self.assertRaises(errors.BadRequestError) as error:
            models.serializer.post_collection(
                self.session, payload, 'posts')

        self.assertEqual(
            error.exception.detail, 'relationships must be an object')
        self.assertEqual(error.exception.status_code, 400)

This test fails because there is no check for this in place in post_collection. Thus when I try to run this test a AttributeError occurs. Specifically the AttributeError is occuring because we are failing to check

if not isinstance(data['data']['relationships'], dict):
    raise BadRequestError('relationships must be an object')

before executing this piece of code in post_collection.

                    data_keys = set(map((
                        lambda x: resource.__jsonapi_map_to_py__.get(x, None)),
                        data['data'].get('relationships', {}).keys()))

Because we do not have this check, a AttributeError occurs because data['data']['relationships'] does not have any keys when trying to execute data['data'].get('relationships', {}).keys() because it is a list.
This is an easy fix that would provide an informative error message to users who send badly formatted payloads.

does not allow the use of custom attribute Column names

In some cases it has been useful for us to override the name of an attribute on a model with the name keyword argument of the Column. However, dump_column is inspecting the __table__ attribute of each model instead of the class itself (see runtime inspection). As the use of Column classes perpetuate through the code a bit I'm not sure how easy a fix for this would be, but as you mentioned you are making significant changes to conform with the RC3 spec I hope this is timely. Am also happy to help out, thanks!

Cheers.

request to add query optimization

Hi Colton,

Thank you for developing this great package. I am very grateful.
Would it be possible to add some query optimization to the get_collection (and get_resource) method based on the included resources? for example, i added the following code to line 621 of serializer.py.

for local in include.keys():
  if local:
      loader = joinedload(local)
      for remote in include[local]:
          if '.' in remote:
              for r in remote.split('.'):
                  loader.joinedload(r)
          else:
              loader.joinedload(remote)
      collection = collection.options(loader)

Thanks,
Omar

Query Parameters not implemented?

Hey there,

I'll take a look at serializer again later this week, but from my side it looks like query parameters aren't implemented. From the spec:

Implementation specific query parameters MUST adhere to the same constraints as member names with the additional requirement that they MUST contain at least one non a-z character (U+0061 to U+007A). It is RECOMMENDED that a U+002D HYPHEN-MINUS, โ€œ-โ€œ, U+005F LOW LINE, โ€œ_โ€, or capital letter is used (e.g. camelCasing).

If a server encounters a query parameter that does not follow the naming conventions above, and the server does not know how to process it as a query parameter from this specification, it MUST return 400 Bad Request.

I'm sending nonsensical query params, and I don't get a 400. Also correctly-formatted QP's don't seem to elicit the right response either.

Incomplete support for SQLAlchemy types

Running into this error with a postgres numeric type which leverages decimal.Decimal under the hood:

TypeError: Decimal('1.6') is not JSON serializable

I think I see how to add it, so I'll put up a PR if I'm successful.

Post_collection is not raising BadRequestError when payload relationships key's value are not a dictionary

It is very important that the library is able to return some sort of response that let's the user know they submitted an invalid request by raising a BadRequestError and returning a 400 and some sort of description explaining why.

Through testing post_collection, I found that we are not raising a BadRequestError when we send a payload where the relationships key's value are not of type dictionary.
For example:

    def test_add_resource_when_payload_relationships_keys_value_are_not_type_dict(self):
        """Create a resource when payload relationships key's are not of type dictionary returns 400.

        Each relationships key's value must be of type dictionary.
        A BadRequestError is raised.
        """
        user = models.User(
            first='Sally', last='Smith',
            password='password', username='SallySmith1')
        self.session.add(user)
        self.session.commit()

        payload = {
            'data': {
                'attributes': {
                    'first': 'Sally',
                    'last': 'Smith',
                    'username': 'SallySmith1',
                    'password': 'password',
                },
                'type': 'users',
                'relationships': {
                    'posts': [{
                        'data': [{
                            'type': 'posts',
                            'id': 1
                        }]
                    }]
                }
            }
        }

        with self.assertRaises(errors.BadRequestError) as error:
            models.serializer.post_collection(
                self.session, payload, 'users')

        self.assertEqual(
            error.exception.detail, 'relationship posts must be an object')
        self.assertEqual(error.exception.status_code, 400)

This test fails because there is no check for this in place in post_collection. Thus when I try to run this test a AttributeError occurs. Specifically the AttributeError is occuring because we are failing to check

if not isinstance(data_rel, dict):
    raise BadRequestError('relationship {0} must be an object'.format(api_key))

before executing this piece of code in post_collection.

                if 'data' not in data_rel.keys():
                    raise BadRequestError(
                        'Missing data key in relationship {}'.format(key))

Because we do not have this check, a AttributeError occurs because just prior to this code we have:

                data_rel = data['data']['relationships'][api_key]

Thus when trying to do data_rel.keys() when data_rel is actually a list, results in an AttributeError since a 'list' object does not have the attribute 'keys'.
This is an easy fix that would provide an informative error message to users who send badly formatted payloads.

Post_collection is not raising BadRequestError when payload relationships data is not array of dictionaries

It is very important that the library is able to return some sort of response that let's the user know they submitted an invalid request by raising a BadRequestError and returning a 400 and some sort of description explaining why.

Through testing post_collection, I found that we are not raising a BadRequestError when we send a payload where the relationships data is not an array of dictionaries.

For example:

def test_add_resource_when_relationships_data_has_invalid_format(self):
        """Create resource when relationships data is not an array of dictionaries returns 400.

        A BadRequestError is raised.
        """
        payload = {
            'data': {
                'attributes': {
                    'first': 'Sally',
                    'last': 'Smith',
                    'username': 'SallySmith1',
                    'password': 'password',
                },
                'type': 'users',
                'relationships': {
                    'posts': {
                        'data': ['foo']
                    }
                }
            }
        }

        with self.assertRaises(errors.BadRequestError) as error:
            models.serializer.post_collection(
                self.session, payload, 'users')

        self.assertEqual(
            error.exception.detail, 'posts data must be an array of objects')
        self.assertEqual(error.exception.status_code, 400)

This test fails because there is no check for this in place in post_collection. Thus when I try to run this test a AttributeError occurs. Specifically the AttributeError is occuring because we are failing to check

if not isinstance(item, dict):
    raise BadRequestError('{} data must be an array of objects'.format(api_key))

at the beginning of the for loop below:

                    for item in data_rel:
                        if not {'type', 'id'} in set(item.keys()):
                            raise BadRequestError(
                                '{} must have type and id keys'.format(key))

Because we do not have this check a AttributeError occurs because the item does not have any keys when trying to execute item.keys().
This is an easy fix that would provide an informative error message to users who send badly formatted payloads.

Getting a TypeError exception on a post_collection call

    def post_collection(self, session, data, api_type):
        """
            Create a new Resource.

            :param session: SQLAlchemy session
            :param data: Request JSON Data
            :param params: Keyword arguments
            """
        model = self._fetch_model(api_type)
        self._check_json_data(data)

        orm_desc_keys = model.__mapper__.all_orm_descriptors.keys()

        if 'type' not in data['data'].keys():
            raise MissingTypeError()

        if data['data']['type'] != model.__jsonapi_type__:
            raise InvalidTypeForEndpointError(
                model.__jsonapi_type__, data['data']['type'])

        resource = model()
        check_permission(resource, None, Permissions.CREATE)

        data['data'].setdefault('relationships', {})
        data['data'].setdefault('attributes', {})

        data_keys = set(map((lambda x: resource.__jsonapi_map_to_py__.get(x, None)), data['data'].get('relationships', {}).keys()))
        model_keys = set(resource.__mapper__.relationships.keys())
        if not data_keys <= model_keys:
            raise BadRequestError(
                '{} not relationships for {}'.format(
                    ', '.join(list(data_keys -
>                                  model_keys)), model.__jsonapi_type__))
E           TypeError: sequence item 0: expected str instance, NoneType found

.tox/py35/lib/python3.5/site-packages/sqlalchemy_jsonapi/serializer.py:968: TypeError

I was attempting to POST an object with a one-to-many related object and hit this exception. I'm sure I'm doing something wrong as I'm new to jsonapi, but I wish it produced a better error message this.

Library not serializing 'id' as string which goes against spec

It is important that the library abides by the JSONAPI spec.

The JSONAPI spec says that
"Every resource object MUST contain an id member and a type member. The values of the id and type members MUST be strings."

Although someone might have the id stored as an integer in the database, it is important that is converted to string during serialization.

To bring this up to spec, we would need to change two functions:

  • _render_full_resource
  • _render_short_instance

The serialization that occurs in _render_full_resource associates the 'id' to the instance.id rather than the str(instance.id).

    def _render_full_resource(self, instance, include, fields):
        """
        Generate a representation of a full resource to match JSON API spec.

        :param instance: The instance to serialize
        :param include: Dictionary of relationships to include
        :param fields: Dictionary of fields to filter
        """
        api_type = instance.__jsonapi_type__
        orm_desc_keys = instance.__mapper__.all_orm_descriptors.keys()
        to_ret = {
            'id': instance.id,
            'type': api_type,
            'attributes': {},
            'relationships': {},
            'included': {}
        }
        ...

The same occurs in _render_short_instance. During serialization, the library 'id' corresponds to the instance.id rather than the str(instance.id).

    def _render_short_instance(self, instance):
        """
        For those very short versions of resources, we have this.

        :param instance: The instance to render
        """
        check_permission(instance, None, Permissions.VIEW)
        return {'type': instance.__jsonapi_type__, 'id': instance.id}

These are easy fixes by simply doing str(instance.id) instead of instance.id.

Unreachable code inside post_collection due to check always evaluating to true

There is a check that is always evaluating to true and it is the following:

if not {'type', 'id'} in set(item.keys()):
                            raise BadRequestError(
                                '{} must have type and id keys'.format(key))

This check happens when we have a test like:

    def test_add_resource_with_invalid_relationships_data(self):
        """Create resource with invalid OneToMany relationship data returns 400.

        In a OneToMany relationship, the value of the data key inside 
        the relationships key must be of type array. 
        A BadRequestError is raised.
        """
        payload = {
            'data': {
                'attributes': {
                    'first': 'Sally',
                    'last': 'Smith',
                    'username': 'SallySmith1',
                    'password': 'password',
                },
                'type': 'users',
                'relationships': {
                    'posts': {
                        'data': [{
                            'type': 'posts',
                        },
                        {
                            'type': 'posts',
                        }]
                    }
                }
            }
        }

        with self.assertRaises(errors.BadRequestError) as error:
            models.serializer.post_collection(
                self.session, payload, 'users')

        self.assertEqual(error.exception.detail, 'posts must have type and id keys')
        self.assertEqual(error.exception.status_code, 400)

In fact if we changed the values inside 'relationships' in the payload we could change it to:

               'relationships': {
                    'posts': {
                        'data': [{
                            'type': 'posts',
                            'id': 1
                        },
                        {
                            'type': 'posts',
                            'id': 2
                        }]
                    }
                }

or

               'relationships': {
                    'posts': {
                        'data': [{
                            'type': 'posts',
                            'id': 1,
                            'title': 'test'
                        },
                        {
                            'type': 'posts',
                            'id': 2,
                            'title': 'dummy'
                        }]
                    }
                }

and we would still evaluate that check as to true.

This is because trying to check if a set is in a set will evaluate to false.
For example if we do

{'type','id'} in {'type','id'} 
{'type', 'id'} in {'id'}
{'type', 'id'} in {'type','id','title'}

These will always return false and since the if statement is checking if not thus it always evaluates to true and the BadRequestError is raised.

Original tests never hit this piece of code since they were not testing OneToMany relationship posts.

A fix to this problem could be to check if not {'type', 'id'} == set(item.keys())

BadRequestError is not being raised succesfully when sending garbage attributes in patch_resource

Looking at the patch_resource function, we have

def patch_resource(self, session, json_data, api_type, obj_id):
        """
        Replacement of resource values.

        :param session: SQLAlchemy session
        :param json_data: Request JSON Data
        :param api_type: Type of the resource
        :param obj_id: ID of the resource
        """
        model = self._fetch_model(api_type)
        resource = self._fetch_resource(session, api_type, obj_id,
                                        Permissions.EDIT)
        self._check_json_data(json_data)
        orm_desc_keys = resource.__mapper__.all_orm_descriptors.keys()

        if not ({'type', 'id'} <= set(json_data['data'].keys())):
            raise BadRequestError('Missing type or id')

        if str(json_data['data']['id']) != str(resource.id):
            raise BadRequestError('IDs do not match')

        if json_data['data']['type'] != resource.__jsonapi_type__:
            raise BadRequestError('Type does not match')

        json_data['data'].setdefault('relationships', {})
        json_data['data'].setdefault('attributes', {})

        missing_keys = set(json_data['data'].get('relationships', {}).keys()) \
            - set(resource.__jsonapi_map_to_py__.keys())

        if missing_keys:
            raise BadRequestError(
                '{} not relationships for {}.{}'.format(
                    ', '.join(list(missing_keys)),
                    model.__jsonapi_type__, resource.id))

        attrs_to_ignore = {'__mapper__', 'id'}

        session.add(resource)

        try:
            for key, relationship in resource.__mapper__.relationships.items():
                api_key = resource.__jsonapi_map_to_api__[key]
                attrs_to_ignore |= set(relationship.local_columns) | {key}

                if api_key not in json_data['data']['relationships'].keys():
                    continue

                self.patch_relationship(
                    session, json_data['data']['relationships'][api_key],
                    model.__jsonapi_type__, resource.id, api_key)

            data_keys = set(map((
                lambda x: resource.__jsonapi_map_to_py__.get(x, None)),
                json_data['data']['attributes'].keys()))
            model_keys = set(orm_desc_keys) - attrs_to_ignore

            if not data_keys <= model_keys:
                raise BadRequestError(
                    '{} not attributes for {}.{}'.format(
                        ', '.join(list(data_keys - model_keys)),
                        model.__jsonapi_type__, resource.id))

            for key in data_keys & model_keys:
                setter = get_attr_desc(resource, key, AttributeActions.SET)
                setter(resource, json_data['data']['attributes'][resource.__jsonapi_map_to_api__[key]])  # NOQA
            session.commit()
        except IntegrityError as e:
            session.rollback()
            raise ValidationError(str(e.orig))
        except AssertionError as e:
            session.rollback()
            raise ValidationError(e.msg)
        except TypeError as e:
            session.rollback()
            raise ValidationError('Incompatible data type')
        return self.get_resource(
            session, {}, model.__jsonapi_type__, resource.id)

Looking at this code we could assume that if we had a test such as:

    def test_patch_resource_unknown_attributes(self):
        user = models.User(
            first='Sally', last='Smith',
            password='password', username='SallySmith1')
        self.session.add(user)
        blog_post = models.Post(
            title='This Is A Title', content='This is the content',
            author_id=user.id, author=user)
        self.session.add(blog_post)
        comment = models.Comment(
            content='This is a comment', author_id=user.id,
            post_id=blog_post.id, author=user, post=blog_post)
        self.session.add(comment)
        self.session.commit()
        payload = {
            'data': {
                'type': 'posts',
                'id': blog_post.id,
                'attributes': {
                     'title': 'This is a new title',
                     'content': 'This is new content',
                     'author-id': 1,
                     'nonexistant': 'test'
                 },
                'relationships': {
                    'author': {
                        'data': {
                            'type': 'users',
                            'id': user.id
                        }
                    }
                }
            }
        }

        with self.assertRaises(errors.BadRequestError) as error:
            models.serializer.patch_resource(
                self.session, payload, 'posts', blog_post.id)

        expected_detail = 'nonexistant not attribute for posts.1'
        self.assertEqual(error.exception.detail, expected_detail)
        self.assertEqual(error.exception.status_code, 400)

That we would actually get a BadRequestError and the correct expected_detail.

However we do not get this because if we look at this specific section of patch_resource

            data_keys = set(map((
                lambda x: resource.__jsonapi_map_to_py__.get(x, None)),
                json_data['data']['attributes'].keys()))
            model_keys = set(orm_desc_keys) - attrs_to_ignore

            if not data_keys <= model_keys:
                raise BadRequestError(
                    '{} not attributes for {}.{}'.format(
                        ', '.join(list(data_keys - model_keys)),
                        model.__jsonapi_type__, resource.id))

the data_keys actually becomes a set of {'title, 'None', 'author_id', 'content'} and the model_keys is {'title', 'author_id', 'content'}

When we try to raise the BadRequestError we then get aTypeError: 'sequence item 0: expected string, NoneType found'.

Thus if you look at the try/except, this is then caught by

      except TypeError as e:
            session.rollback()
            raise ValidationError('Incompatible data type')

So although some sort of error is raised and it is gracefully handled. It is not the expected error since the TypeError is occuring when trying to raise that BadRequestError and is thus excepted and instead a ValidationError('Incompatible data type') is raised.

I feel that the actually BadRequestError should be raised correctly so that we have a more informed error as to what happened rather than a ValidationError.

Optional Eager Loading

Eager Loading of relationships should be optional. This way queries can be limited and generation time can be reduced.

So Colton of the future where you aren't tired, implement this.

raising an error class does not print the error details

In a shell:

import errors
raise errors.ValidationError("foo")
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
<ipython-input-11-464b5e215eb8> in <module>()
----> 1 raise errors.ValidationError("foo")

ValidationError:

This is because the error classes do not call the super constructor in their constructor overwrites. Do you have a reason to not call the super constructors there? I have fixed this locally so I can see what is actually going on when, e.g. a ValidationError was raised. When you are interested I can open a pull request with my fix.

TypeError occurs when formatting detail when raising BadRequestError with garbage relationships in post_collection

Here is a test that should return a BadRequestError:

    def test_add_resource_with_invalid_relationships(self):
        """Create resource with unknown OneToMany relationship key returns 400.

        A BadRequestError is raised.
        """
        payload = {
            'data': {
                'attributes': {
                    'first': 'Sally',
                    'last': 'Smith',
                    'username': 'SallySmith1',
                    'password': 'password',
                },
                'type': 'users',
                'relationships': {
                    'posts': {
                        'data': [{
                            'type': 'posts',
                            'id': 1
                        },
                        {
                            'type': 'posts',
                            'id': 2
                        }]
                    },
                    'dogs': {
                        'data': [{
                            'id': 1,
                            'type': 'dogs'
                        }]
                    }
                }
            }
        }

        with self.assertRaises(errors.BadRequestError) as error:
            models.serializer.post_collection(
                self.session, payload, 'users')

        self.assertEqual(error.exception.detail, 'dogs not relationship for users')
        self.assertEqual(error.exception.status_code, 400)

However this test will never actually return this detail message because when this check happens:

         data_keys = set(map((
            lambda x: resource.__jsonapi_map_to_py__.get(x, None)),
            data['data'].get('relationships', {}).keys()))
        model_keys = set(resource.__mapper__.relationships.keys())
        if not data_keys <= model_keys:
            raise BadRequestError(
                '{} not relationships for {}'.format(
                    ', '.join(list(data_keys -
                                   model_keys)), model.__jsonapi_type__))

When we take the difference between data_keys and model_keys we are getting {None} because
data_keys = {'posts',None'} and model_keys = {'posts','comments',logs'}.

When the BadRequestError is raised we get a TypeError: sequence 0: expected str instance, NoneType found.

This happens on both python3 and python2.
I will look at more as to how to fix this.

FlaskJSONAPI doesn't connect to postgres

using a flask shell manager I see this:

db.engine
Engine(postgres://localhost:5432/opengov)
api.serializer.prefix
'http://localhost:5000/api'
app.config['SERVER_NAME']
'localhost:5000'
app.config['SQLALCHEMY_DATABASE_URI']
'postgres://localhost:5432/opengov'`

My init_db manager command initalizes the tables just fine.

Everything seems to be configured properly, yet I still get this error when I hit http://localhost:5000/api/citizen (that's the name of my sqlalchemy table/orm; I chose not to inflect it for the sake of simplicity while trying to figure out how your library works):

{ "errors": [ { "id": "51a8b3de-b1fc-41b9-b1c5-810f77a426d7", "status": 404, "detail": "This backend has not been configured to handle resources of type citizen.", "title": "Resource Type Not Found", "code": "resource_type_not_found" } ] }

Any idea what's off here?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.