EmbeddedDocumentField throws exception in serializer about django-rest-framework-mongoengine

[umutbozkurt]

Mongoengine Serializer does not support EmbeddedDocument, or ListField, yet.
Instead, the serializer uses depth for deciding whether reference(with pk) or embed Documents.
As you see on https://pythonhosted.org/django-rest-framework-mongoengine/#usage ,
you should set your ReferenceFields on model and also your depth properly.

For your case, you should declare EmbeddedDescriptionsReference as Document and set depth at least 1 (consider your JSON depth level)

Hope that clarifies your issue

[mkhndrsn]

Thanks. I don't think that will work in my case because the embedded document is part of the parent document, not a document in a different collection. With the suggested usage in the documentation I received the following error when querying from the db:

id must be an instance of (bytes, str, ObjectId), not <class 'dict'>

[umutbozkurt]

Couldn't think of anything to overcome querying issue, could you post querying code?
Also, I would appreciate any pull requests regarding serializing ListFields

[mkhndrsn]

I get the error when I try to query from the database:

Products.objects.get(id=...)

This is if I change my code to the following:

class EmbeddedDescriptionsReference(Document):
    long_desc = StringField()
    short_desc = StringField()

    meta = {'allow_inheritance': False}


class Products(DynamicDocument):
    descriptions = ReferenceField(EmbeddedDescriptionsReference)

    meta = {'collection': 'products'}


class ProductsSerializer(rest_framework_mongoengine.serializers.MongoEngineModelSerializer):
    class Meta:
        model = Products
        fields = ('descriptions',)

This would be if the documents in the products collection are in the following form:

{
    '_id': ObjectId("..."),
    'descriptions': {
        'long_desc': 'Long description',
        'short_desc': 'Short description'
    }
}

Yeah I'll take a look at the code and see if I have any ideas for embedded documents and lists

[umutbozkurt]

There is something odd on DynamicDocument, I changed Products inherit Document.

class EmbeddedDescriptionsReference(Document):
    long_desc = StringField()
    short_desc = StringField()

class Products(Document):
    descriptions = ReferenceField(EmbeddedDescriptionsReference)

class ProductSerializer(MongoEngineModelSerializer):
    class Meta:
        model = Products
        depth = 1
        related_model_validations = {'descriptions': EmbeddedDescriptionsReference}

The key point to understand at mongoengine serializer is, to save foreign keys to collections -as object id- and fetch them as documents.

When you save Products, it looks like that on DB

{
    "_id" : ObjectId("536a3c6d571a6506e110837c"),
    "descriptions" : ObjectId("536a3bff571a6506e110837b"),
        "some_other_field":"the value"
}

So, when querying, you should write the line like this:

product = Products.objects.get(pk=the_pk) 

Use object IDs to query, not the whole Document.
Im guessing you are using the whole document to query, thats why you get that exception

The limitations are:

• You cant use EmbeddedDocument inside some other Document without ReferenceFields
• You should save your EmbeddedDescriptionsReference before saving your Product

I'm pretty busy nowadays, but I will make some enhancements to mongoengine serializer and develop a 3rd party what uses all features of mongoengine, starting EmbeddedDocuments first

[mkhndrsn]

Thanks. I'm querying the products collection by the primary key or some other string/int field. The error is a result of the descriptions field being a sub-document in the database, rather than an ObjectId. I don't have control over the document structure so I can't change the descriptions field to an ObjectId that references a different collection.

Even if I could, I wouldn't want to change it, since MongoDB is designed to embed sub-documents rather than reference other collections. Ideally, I would get the results in one query per HTTP GET. When referencing other collections, the client application must make a separate query for each reference and this can degrade performance. If an application must treat each document as a flat document and only using references for sub-documents, it would be better off using a relational database so that joins can be done within the database.

I will take a look when I have some free time. I think adding additional logic in to_native() in MongoEngineModelSerializer may work. Also, I don't think the depth should apply to embedded documents. Any thoughts on that?

[umutbozkurt]

If I can manage to serialize EmbeddedDocument without treating it as a Document(like a seperate Document as you mentioned), depth will not be necessary for the serializer anyway. The only reason I'm using depth is to make decision whether embed or reference the Document. And with ListField support, MongoEngineModelSerializer would be a nice 3rd party.

For now, behaviour like ListField is supported via custom SerializerMethodField

Finally, adding logic in to_native() may work with some built-in django model field extensions

[umutbozkurt]

@mkhndrsn ,
Check out embedded_document_serializing branch and let me know if it works for EmbeddedDocuments. It looks OK to me.

Any thoughts on further development (of serializing EmbeddedObjects)?

[mkhndrsn]

Ok, thanks. I'll take a look. Right now I'm thinking maybe requiring a a serializer for each embedded document type and then adding that serializer as a field similar to this:

class CommentSerializer(serializers.Serializer):
    user = UserSerializer()
    content = serializers.CharField(max_length=200)
    created = serializers.DateTimeField()

My current project is just finishing up so I should have some time next week or the week after to look into this further

[umutbozkurt]

Ok nice, I'll try nested serializers asap

[zzzuzik]

@umutbozkurt

Am I correct, that there is no way that Serializer would work with this model?
EmbeddedDocument -> ListField(EmbeddedDocumentField())

class Comment(EmbeddedDocument):
    content = StringField()
    name = StringField(max_length=120)

class Post(Document):
    title = StringField(max_length=120, required=True)
    author = ReferenceField(User)
    tags = ListField(StringField(max_length=30))
    comments = ListField(EmbeddedDocumentField(Comment))

Thanks a lot

[umutbozkurt]

I am working on it, take a look at list field branch and embedded document branch, it is possible

[zzzuzik]

Cheers!
I've already checked out both branches before asking. Based on your judgement, how far it is from implantation? I mean, should I go through the code and try my best to get it done by myself. Is it the whole new code refactoring, or just fixing/adding some methods/parameters?

Thank you

[umutbozkurt]

Its just adding some methods, of course you can do it by yourself.
But the list field branch is not truly complete, yet.

I would appreciate any contribution though. I'll try to finalize and merge branches asap

[zzzuzik]

Thank you

[umutbozkurt]

@zzzuzik check out latest commit

[zzzuzik]

@umutbozkurt wow! really appreciate such a fast reaction. Thanks a million!

I had to replace unicode with str to make it work in Py3;
otherwise, NameError at <> name 'unicode' is not defined

I am going to keep using your latest code, so in case of any surprising findings I will report.
And thanks again for your efforts