gameboy86 / matchbox-orm

ORM package for google Cloud Firestore.

Matchbox

Details	Matchbox is orm package for Google Firestore.
Repository	https://github.com/gameboy86/matchbox
Author	Maciej Gębarski (https://github.com/gameboy86)
Contact	[email protected]
License	MIT License
Version	0.2.7

Details

Matchbox is a Python Object-Relational Mapper for Google Firestore.

Installing

 pip install matchbox-orm

Usage

Connect to Firestore

More info, how to generate JSON file with private key you will find on Get started with Cloud Firestore

from matchbox import database

database.db_initialization('path/to/serviceAccount.json')

Model

Create

from matchbox import models

class Test(models.Model):
    age = models.IntegerField()
    name = models.TextField()

    def __unicode__(self):
        return self.id

>> t = Test()
>> print(t)
<Test: e7aad1ec1aa449d2b53b7ca8f2853ea0>

By default all fields are required (except IDField, ReferenceField). This behavior can be change using attributes blank or default.

If we now save model we get:

>> t.save()
AttributeError: Field age required value
>> t.age = 18
>> t.save()
AttributeError: Field name required value
>> t.name = 'Name'
>> t.save()

Another way to create model is use manager create method:

>> Test.objects.create(name='Test', age=29)
<Test: 33eba5fd53244e38aa1b4951f104ec3c>

By default collection name in DB will be create based on model name. If you want to change it, you can do it using Meta. For example:

from matchbox import models

class Test(models.Model):
    age = models.IntegerField()
    name = models.TextField()

    class Meta:
        collection_name = 'TestCollection'

    def __unicode__(self):
        return self.id

>> Test._meta.collection_name
'TestCollection'

Update

Document can be update by two ways: override or update. Example below will override whole document:

>> t = Test.objects.get(id='eba5fd53244e38aa1b4951f104ec3c')
>> t.age = 53
>> t.save()

If we want update only specific fields, we can use update_fields parameter in save method:

>> t = Test.objects.get(id='eba5fd53244e38aa1b4951f104ec3c')
>> t.age = 32
>> t.save(update_fields=['age'])

Fields

Available fields:

• IDField
• IntegerField
• TextField
• TimeStampField
• BooleanField
• ListField
• MapField
• GeoPointField
• ReferenceField

Attributes

Available attributes for all fields:

• blank (If True empty fields will save null in DB.)
• default (If field is empty, on the save, default value will be used. If default value callable it will be called)
• column_name (Name of field in DB. If empty, name of field will be used)

TextField accept on more attribute max-length.

class Test2(models.Model):
    age = models.IntegerField(default=25)
    name = models.TextField(blank=True)

>> t = Test2()
>> t.save()
>> t = Test2.objects.get(id=t.id)
>> print(t.age, t.name)
25 None

IDField

IDField is create automatically by orm. We can't add own, because Firestore doesn't allow for self named id field.

>> t._meta.fields
{
    'age': <matchbox.models.fields.IntegerField at 0x111723f98>,
    'name': <matchbox.models.fields.TextField at 0x111723b70>,
    'id': <matchbox.models.fields.IDField at 0x1117232b0>
}

If you want you can specify your own id:

>> t = Test(age=33, name='test', id='My OWN ID')
>> t.save()
>> t.id
'My OWN ID'

If you change id and save, new document will be create in Firestore.

TimeStampField

class TimeStampFieldExample(models.Model):
    datetimestamp = models.TimeStampField()

    def __unicode__(self):
        return self.id

>> TimeStampFieldExample.objects.create(datetimestamp=datetime.datetime.now())
<TimeStampFieldExample: xp4LHczLwzcpC8Q4yF5s>

>> list(TimeStampFieldExample.objects.filter(datetimestamp__lte=datetime.datetime.now()))
[<TimeStampFieldExample: xp4LHczLwzcpC8Q4yF5s>]

>> TimeStampFieldExample.objects.filter(datetimestamp__lte=datetime.datetime.now()).get().datetimestamp
datetime.datetime(2019, 5, 4, 16, 42, 34, 583953, tzinfo=datetime.timezone(datetime.timedelta(0), '+00:00'))

TimeStampField with callable default

class DefaultTimeStampFieldExample(models.Model):
    created_at = models.TimeStampField(default=datetime.datetime.now)

    def __unicode__(self):
        return self.id

>> tsf = TimeStampFieldExample.objects.create()
>> print(tsf)
<DefaultTimeStampFieldExample: wqAVap5rYW7Zl0cgO9UI>

>> print(tsf.created_at)
2019-11-07 08:30:10.884238+00:00

ListField

class ListFieldExample(models.Model):
    list_f = models.ListField()

    def __unicode__(self):
        return self.id

>> ListFieldExample.objects.create(list_f=[1, 2, 3, 4, 5])
>> list(ListFieldExample.objects.filter(list_f__contains=5))
[<ListFieldExample: vZvDWm2EG6Di1wm85uD8>]

>> ListFieldExample.objects.filter(list_f__contains=5).get().list_f
[1, 2, 3, 4, 5]

MapField

class MapFieldExample(models.Model):
    map_f = models.MapField()

    def __unicode__(self):
        return self.id

>> MapFieldExample.objects.create(map_f = {'a': 1, 'b': 2, 'c': {'a': 1}})
<MapFieldExample: JVggchyQn19knDfx2SNX>

>> list(MapFieldExample.objects.filter(map_f__c__a=1))
[<MapFieldExample: JVggchyQn19knDfx2SNX>]

>> list(MapFieldExample.objects.filter(map_f__c__a=1))[0].map_f
{'b': 2, 'c': {'a': 1}, 'a': 1}

GeoPointField

To save GeoPoint data you must use class GeoPointValue

class GeoPointFieldExample(models.Model):
    geo_point_f = models.GeoPointField()

    def __unicode__(self):
        return self.id

>> gpf = GeoPointFieldExample()
>> gpf.geo_point_f = GeoPointValue(latitude=52.2297, longitude=21.0122)
>> gpf.save()

>> list(GeoPointFieldExample.objects.all())[0].geo_point_f
<matchbox.models.utils.GeoPointValue at 0x11191da58>

>> list(GeoPointFieldExample.objects.all())[0].geo_point_f.latitude
52.2297

ReferenceField

One of field offered by FireStore is Reference. In one document you can store reference to another document.

class User(models.Model):
    name = models.TextField()

    def __unicode__(self):
        return self.id

class Class(models.Model):
    name = models.TextField()
    user = models.ReferenceField(User)

    def __unicode__(self):
        return self.id

>> u = User.objects.create(name='Alex')
>> c = Class.objects.create(name='A1', user=u)
>> c.user
<User: cdda43cf3d65413f9eea17349e8222b8>

>> c.user.id, c.user.name
('cdda43cf3d65413f9eea17349e8222b8', 'Alex')

Query

objects.get

    class User(models.Model):
    name = models.TextField()

    def __unicode__(self):
        return self.id

>> u = User.objects.create(name='Alex')
>> User.objects.get(id=u.id)
<User: fe500b4bc341471fa3118854b705c674>

objects.all

Return all documents in collection

class User(models.Model):
    name = models.TextField()

    def __unicode__(self):
        return self.id

class Class(models.Model):
    name = models.TextField()
    user = models.ReferenceField(User)

    def __unicode__(self):
        return self.id

>> User.objects.create(name='Tom')
>> User.objects.create(name='Alex')
>> User.objects.create(name='Michael')
>> User.objects.all()
<matchbox.queries.queries.FilterQuery at 0x1116a3978>

>> list(User.objects.all())
[<User: 6b8e2190ebe3428e8c30433e74287639>,
<User: 96767fdc81ba48779683868d2a81cbba>,
<User: fe500b4bc341471fa3118854b705c674>]

objects.filter

Filter is based on django filter method. FireStore allow following comparison, with are mapped to:

FireStore	Matchbox
<	lt
<=	lte
>	gt
>=	gte
==	not need
array_contains	contains

class User(models.Model):
    name = models.TextField()
    evaluations = models.ListField()
    age = models.IntegerField(default=20)

    def __unicode__(self):
       return self.id

>> User.objects.create(name='Tom', evaluations=[1,1,2], age=15)
>> User.objects.create(name='Michael', evaluations=[2,3,5])
>> User.objects.create(name='Michael', evaluations=[4,4,2])
>> User.objects.filter()
[<User: 2dce37628c4345b0a9d1a721265984b4>,
<User: 348bf6888d1e4d22afd29385f8c1a330>,
<User: 389ac1ca88614d5fa5e53facb1249576>]

>> User.objects.filter(age__gte=10, age__lte=15)
[<User: 348bf6888d1e4d22afd29385f8c1a330>]

>> u = User.objects.filter(age__gte=10, age__lte=15).get()
>> print(u.age)
15

>> list(User.objects.filter(name='Michael'))
[<User: 2dce37628c4345b0a9d1a721265984b4>,
<User: 389ac1ca88614d5fa5e53facb1249576>]

>> list(User.objects.filter(name='Michael').filter(evaluations=[4,4,2])) # or list(User.objects.filter(name='Michael', evaluations=[4,4,2]))
[<User: 2dce37628c4345b0a9d1a721265984b4>]

>> u = User.objects.filter(name='Michael', evaluations=[4,4,2]).get()
>> print(u.id, u.age, u.name, u.evaluations)
2dce37628c4345b0a9d1a721265984b4 20 Michael [4, 4, 2]

>> list(User.objects.filter(evaluations__contains=3))
[<User: 389ac1ca88614d5fa5e53facb1249576>]

>> u = User.objects.filter(evaluations__contains=3).get()
>> u.id, u.name, u.evaluations
('389ac1ca88614d5fa5e53facb1249576', 'Michael', [2, 3, 5])

You can also filter by ReferenceField

class Class(models.Model):
    name = models.TextField()
    user = models.ReferenceField(User)

    def __unicode__(self):
        return self.id

>> c = Class.objects.create(name='A1', user=User.objects.all().get())
>> c.user.id, c.user.name
'2dce37628c4345b0a9d1a721265984b4', 'Michael'

>> Class.objects.filter(user=u).get()
<Class: c3728ca35d25414794f6071d3acb3e2b>

order_by and limit

>> [(u.age, u.name) for u in User.objects.all()]
[(20, 'Michael'), (15, 'Tom'), (20, 'Michael')]

>> [(u.age, u.name) for u in User.objects.all().order_by('age')]
[(15, 'Tom'), (20, 'Michael'), (20, 'Michael')]

>> [(u.age, u.name) for u in User.objects.all().order_by('-age')]
[(20, 'Michael'), (20, 'Michael'), (15, 'Tom')]

>> [(u.age, u.name) for u in User.objects.all().order_by('-age').limit(2)]
[(20, 'Michael'), (20, 'Michael')]

Paginate

from matchbox.queries.paginator import Paginator

class User(models.Model):
    name = models.TextField()
    age = models.IntegerField()

    def __unicode__(self):
        return self.id


>> pag = Paginator(User.objects.filter(age__gte=10), 100)
>> for q_data in pag:
     print([x.name for x in q_data])  # make request for 100 documents per loop

Delete

We can delete document by instance or by filter.

>> u = User.objects.all().get()
>> u.delete()

>> User.objects.filter(name='Alex').delete()

Delete whole collection:

>> User.objects.delete()
or
>> User.objects.filter().delete()

Managers

Like in Django we can create own Managers. For example:

class User(models.Model):
    name = models.TextField()
    evaluations = models.ListField()
    age = models.IntegerField(default=20)

    def __unicode__(self):
        return self.id

class AManager(models.Manager):
    def get_queryset(self):
        return super().get_queryset().filter(active=True)


class DManager(models.Manager):
    def get_queryset(self):
        return super().get_queryset().filter(active=False)


class Class(models.Model):
    name = models.TextField()
    user = models.ReferenceField(User)
    active = models.BooleanField()

    a_objects = AManager()
    f_objects = DManager()

    def __unicode__(self):
        return self.id

>> c1 = Class.objects.create(active=True, name='DD21')
>> c2 = Class.objects.create(active=True, name='DD22')
>> c3 = Class.objects.create(active=False, name='CC22')
>> c4 = Class.objects.create(active=False, name='CC11')
>> list(Class.objects.all())
[<Class: 96Ww50qJVh53v46iyOPP>,
 <Class: cjGlGWM8RiJqcAQLGvXK>,
 <Class: pgvWsXY47GrYO4Eiyp2W>,
 <Class: vHZMVjda2wNEVDmoxTe2>]

>> list(Class.f_objects.all())
[<Class: pgvWsXY47GrYO4Eiyp2W>, <Class: vHZMVjda2wNEVDmoxTe2>]

>> list(Class.a_objects.all())
[<Class: 96Ww50qJVh53v46iyOPP>, <Class: cjGlGWM8RiJqcAQLGvXK>]

Abstract model

Abstract model useful when you want to put some common information into a number of other models. You must create base class and add abstract = True in the Meta model class.

For example:

from matchbox import models as fsm, database

database.db_initialization('xxx.json')


class SuffixFsm(fsm.Model):
    createdAt = fsm.TimeStampField()
    createdBy = fsm.TextField(max_length=30, default='')

    class Meta:
        abstract = True


class SystemMaster(SuffixFsm):
    systemName = fsm.TextField(max_length=50, default='')

>> master = SystemMaster(
        systemName='name',
        createdAt=datetime.now(),
        createdBy='test',
    )
>> master.save()
>> master.__dict__

{'id': '9ZCOPU8KRwUB4rRVF1kZ',
 'systemName': 'name',
 'createdAt': datetime.datetime(2019, 7, 4, 21, 36, 56, 472744),
 'createdBy': 'test'}

SubCollections

Let say we want store structure like below in firestore

    (C) rooms
        (D) roomA
        name : "my chat room"
            (C) messages
                (D) message1
                from : "alex"
                msg : "Hello World!"

            (C) message2
                ...

        (D) roomB
            ...

(C) -> Collection
(D) -> Document

from matchbox import models, database
database.db_initialization('xxx.json')

class Message(models.Model):
    by = models.TextField()
    msg = models.TextField()

    class Meta:
        collection_name = 'messages'


class Room(models.Model):
    name = models.TextField()

    class Meta:
        collection_name = 'rooms'

To create subcollection in document, we must set path in model to document using set_base_path.

ModelClass.set_base_path(model_instance) -> model_instance must be stored in firestore before passed to method.

>> r = Room.objects.create(name='roomA')
>> Message.set_base_path(r)
>> # Wrong Room(name='roomA'); Message.set_base_path(r)
>> Message.objects.create(by='Alex', msg='Hello')
>> Message.objects.create(by='Alex', msg='How are you ?')

>> r = Room.objects.create(name='roomB')
>> Message.set_base_path(r)
>> Message.objects.create(by='Neo', msg='Matrix ?')
>> Message.objects.create(by='Matrix', msg='Follow the white rabbit')

IMPORTANT: Default path is '/<collection_name>', so if you don't set path your document will be created in root path. You always can restore default path using ModelClass.reset_base_path().

To check path instance use model_path

>> r = Room.objects.get(name='roomA')  # r.id == 'K8imB6eui5ibfSEZon3e'
>> print(r.model_path)
('rooms', 'K8imB6eui5ibfSEZon3e')

>> r = Room.objects.get(name='roomB')  # r.id == '4QIk9Q5LCrkVz1bWir6w
>> print(r.model_path)
('rooms', '4QIk9Q5LCrkVz1bWir6w')

>> Message.set_base_path(r)
>> m = Message.objects.get(by='Neo')  # m.id == 'YypGDFPi5M1NYeWqROSq'
>> print(m.model_path)
('rooms', '4QIk9Q5LCrkVz1bWir6w', 'messages', 'YypGDFPi5M1NYeWqROSq')

To check model path use 'path' property

>> print(Room.path)  # ('rooms', )

To get all messages from 'roomB' filtered by 'by' field:

>> r = Room.objects.get(name='roomB')
>> Message.set_base_path(r)
>> neo_messages = Message.objects.filter(by='Neo')
>> print(len(list(neo_messages)))
1

>> matrix_messages = Message.objects.filter(by='Matrix')
>> print(len(list(matrix_messages)))
1

Now let say, we want to delete all messages in 'roomA':

>> r = Room.objects.get(name='roomA')
>> Message.set_base_path(r)
>> print(len(list(Message.objects.all())))
2

>> Message.objects.delete()
>> print(len(list(Message.objects.all())))
0

>> r = Room.objects.get(name='roomB')
>> Message.set_base_path(r)
>> print([x.msg for x in Message.objects.all()])
['Follow the white rabbit', 'Matrix ?']

IMPORTANT: We can't delete room (Room.objects.get(name='roomA').delete()). If we do this in this way, references in firestore to messages will still exist. So before deleting Collection, make sure you delete all subcollections independently from his documents.

matchbox-orm's Issues

(Feature) Support sub-collections

Currently, collection names are static and root level. I want to make a get query on a sub-collection of a model.

Querying could look something like this:

class A(models.Model):
    a_field_1 = models.TextField()

class B(models.Model):
    b_field_1 = models.TextField()
    b_field_2 = models.TextField()

a = A.objects.get(a_field_1='value1')
b = B.objects.get(parent_model=a, b_field_1='value1', b_field_2='value2')

Internally, the full collection name for 'a' and its own unique id would be appended to the beginning of the collection name for 'b' of class B. Something like this:

full collection name for 'a': a
full collection name for 'b': a/<unique_id_for_a>/b

This format could be used for any depth of sub-collections

Default value for Model.ListField does not get set

I have ran into an issue where I set the 'default' param for the subclass of a Model object to be an empty list, however it does not get set when I create a new object of that subclass with no params.

Default values should be filled into the field so that they can be used right away, before new model objects are even saved.

.objects.all() skips objects without a field

Hi,
Not sure if this is expected behavior or a bug but figured out I'd ask. I tried to call .objects.all() fetching objects. One of the fields is a TimeStampField, but only some objects have the value assigned to it in the Firebase (it was added later, when already some records were in database).

When calling .objects.all(), all objects without a value for TimeStampField are skipped.

Any idea if I can somehow get all objects, even those that don't have this field?

Dzięki za pomoc!

ReferenceField does not return a proper DocumentReference db value

I noticed that ReferenceField.db_value no longer returns any value, where it previously returned a DocumentReference object.

I have needed to do something like this in order to filter by reference:

class ReferenceField(Field):
...
    def db_value(self, value):
        ...
        return google.cloud.firestore_v1.document.DocumentReference(
            value.collection_name(), value.id, client=db.conn
        )
...

This has allowed me to filter by reference with a full collection name (which I currently hack fix a certain sub-collection)

Inheritance of abstract model

Hello,

I tried an inheritance of abstract model, but it seemed like it failed.

My code is attached below. When executing it, it's only "systemNumber" and "systemName" which have proper values on the Firestore. The fields of "createdAt" and "createdBy" from "Suffix_fsm" don't appear there.

from matchbox import models as fsm, database
from datetime import datetime
database.db_initialization('xxx.json')

class Suffix_fsm(fsm.Model):
    createdAt = fsm.TimeStampField()
    createdBy = fsm.TextField(max_length=30, default='')

    class Meta:
        abstract = True

class SystemMaster(Suffix_fsm):
    systemNumber = fsm.IDField()
    systemName = fsm.TextField(max_length=50, default='')

master = SystemMaster(
    systemNumber=1,
    systemName='name',
    createdAt=datetime.now(),
    createdBy='test',
)
master.save()

It would be great if this issue is resolved.

Thank you !

startAt() and startAfter() support

Hi, do you have any plans to support startAt(), endAt(), startAfter() query cursors or have a workaround to do pagination of the results? Thanks!

Can't filter on ReferenceField

I have an 'Attempt' Model that includes a ReferenceField to 'Challenge' Model. When I attempt to filter Attempt based on Challenge, I get an error, that appears to be coming from the Firestore library.

Part of Stack Trace:

File "/home/djwhyte/workspace/BBBBCSegmentTracker/main.py", line 47, in attempts
    atts = Attempt.objects.filter(challenge=c).get()
  File "/home/djwhyte/workspace/BBBBCSegmentTracker/venv/lib/python3.6/site-packages/matchbox/queries/queries.py", line 113, in get
    res = list(self.execute())
  File "/home/djwhyte/workspace/BBBBCSegmentTracker/venv/lib/python3.6/site-packages/matchbox/queries/queries.py", line 100, in execute
    for d in self.raw_execute() if d
  File "/home/djwhyte/workspace/BBBBCSegmentTracker/venv/lib/python3.6/site-packages/matchbox/queries/queries.py", line 84, in raw_execute
    return self.make_query().stream()
  File "/home/djwhyte/workspace/BBBBCSegmentTracker/venv/lib/python3.6/site-packages/matchbox/queries/queries.py", line 70, in make_query
    bsq = bsq.where(*w)
  File "/home/djwhyte/workspace/BBBBCSegmentTracker/venv/lib/python3.6/site-packages/google/cloud/firestore_v1/collection.py", line 250, in where
    return query.where(field_path, op_string, value)
  File "/home/djwhyte/workspace/BBBBCSegmentTracker/venv/lib/python3.6/site-packages/google/cloud/firestore_v1/query.py", line 272, in where
    value=_helpers.encode_value(value),
  File "/home/djwhyte/workspace/BBBBCSegmentTracker/venv/lib/python3.6/site-packages/google/cloud/firestore_v1/_helpers.py", line 200, in encode_value
    "Cannot convert to a Firestore Value", value, "Invalid type", type(value)
TypeError: ('Cannot convert to a Firestore Value', <Challenge: 9zcSTiUbeOe5bNjqXV3d>, 'Invalid type', <class 'model.Challenge'>)

The pertinent area of my Models:

class BaseModel(models.Model):
    audit_inserted = models.TimeStampField()

    class Meta:
        abstract = True

    def __unicode__(self):
        return self.id

    @abstractmethod
    def add(self):
        pass


class Challenge(BaseModel):
    date_from = models.TimeStampField()
    date_to = models.TimeStampField()
    segment_id = models.IntegerField()
    segment_name = models.TextField()
    audit_inserted = models.TimeStampField(datetime.now())

    @staticmethod
    def add(segment, year, month):
        """Takes a stravalib.model.Segment object and adds it as the target challenege to cover the given month of the given year"""
        _, end_day = calendar.monthrange(year, month)

        date_from = datetime.combine(date(year, month, 1), datetime.min.time())
        date_to = datetime.combine(date(year, month, end_day), datetime.min.time())

        c = Challenge.objects.create(segment_id=segment.id, segment_name=segment.name, date_from=date_from,
                                     date_to=date_to, audit_inserted=datetime.utcnow())
        c.save()

        return c


class Attempt(BaseModel):
    member = models.ReferenceField(Member)
    challenge = models.ReferenceField(Challenge)
    recorded_time_secs = models.IntegerField()
    activity_timestamp = models.TimeStampField()
    activity_id = models.IntegerField(blank=True)

    @staticmethod
    def add(effort, member, challenge):
        a = Attempt.objects.create(id=effort.id, member=member, challenge=challenge,
                                   recorded_time_secs=effort.elapsed_time.total_seconds(),
                                   activity_timestamp=effort.start_date_local, activity_id=effort.activity.id,
                                   audit_inserted=datetime.utcnow())
        a.save
                     member_id=member.id, challenge_id=challenge.id, attempt_id=a.id)

        return a

The code that is making the call:

        c = Challenge.objects.get(id=challenge)
        print (c)
        atts = Attempt.objects.filter(challenge=c).get()

Note that the 'print (c)' above is correctly outputting the Challenge object that is correctly referenced: <Challenge: 9zcSTiUbeOe5bNjqXV3d>

Notable parts of requirements.txt:

google-cloud-firestore==1.4.0
matchbox-orm==0.2

How can make my model can inherit?

Hi,
in my project, I used similar structures with multiple classes,
such as,

class UserDatabase(models.Model):
    class Meta:
        collection_name = 'User'
    birth = models.TimeStampField()
    email = models.TextField()
    name = models.MapField()

    def __unicode__(self):
        return self.id

    def to_dict(self):
        return self.__dict__

class AnnotatorDatabase(models.Model):
    class Meta:
        collection_name = 'annotators'
    total_annotation = models.IntegerField()

    def __unicode__(self):
        return self.id

    def to_dict(self):
        return self.__dict__

So I want to make above codes to like below.

class BaseDatabase(models.Model):

    def __unicode__(self):
        return self.id

    def to_dict(self):
        return self.__dict__

class Userdatabase(BaseDatabase):
    birth = models.TimeStampField()
    email = models.TextField()
    name = models.MapField()

    class Meta:
        collection_name = 'User'

class AnnotatorDatabase(BaseDatabase):
    total_annotation = models.IntegerField()

    class Meta:
        collection_name = 'Annotator'

but it shows error, AttributeError: Can't inherit from non abstract class so coud you give me any tip for the better design?

Thank you.