.objects.all() skips objects without a field about matchbox-orm

[gameboy86]

Hi,
i can't reproduce your issue.

Steps which i make to reproduce your problem:
1.) Create without TimeStamField:

class Name(models.Model):
    name = models.TextField()

n = Name.objects.create(name='AAAA')
list(Name.objects.all())
[<__main__.Name at 0x10f9fdf98>]

2. I added TimeStampField and created second row

class Name(models.Model):
    name = models.TextField()
    date = models.TimeStampField()

n = Name.objects.create(name='AAAA', date=datetime.datetime.now())
list(Name.objects.all())
[<__main__.Name at 0x10faa4668>, <__main__.Name at 0x10faa4630>]

How you see, it return all rows. Can you write (in steps) how you made this ?

[jwitos]

I'm sorry, I wasn't being precise enough. I see now that the problem is not with objects.all() but in sorting: objects.all().order_by('added_datetime').

Reproduction would be:

1. In Firebase, in my Collection there are three Documents:
   Document 1: name=Object1, dttm_created=1 January 2020 01:01:01
   Document 2: name=Object2, (no dttm_created field)
   Document 3: name=Object3, (no dttm_created field)

2. Fetch all documents without ordering

class Name(models.Model):
    name = models.TextField()
    dttm_created = models.TimeStampField(blank=True)
  
docs = Name.objects.all()
print(len(list(docs))) # Returns all 3 documents

3. Fetch all documents with ordering

class Name(models.Model):
    name = models.TextField()
    dttm_created = models.TimeStampField(blank=True)
  
docs = Name.objects.all().order_by('dttm_created')
print(len(list(docs))) # Returns only one document

Again, not sure if that's expected behavior, and I'm able to sort the results after getting them from the Firebase, it would just be nice to fetch all of them at once.

[gameboy86]

Hi i check and this is firebase behavior. Problem is that when you create document without dttm_created, key wasn't added to document in FireBase. You can check this:

>> from matchbox.database import db_initialization, db
>> db_initialization(os.environ['FIRESTORE'])
>> raw_from_firebase = db.conn.collection('name').order_by('date').stream()

>> [x.__dict__ for x in raw_from_firebase]
>> [
{'_reference': <google.cloud.firestore_v1.document.DocumentReference at 0x10fb10c88>,
  '_data': {'name': 'AAAA',
   'date': DatetimeWithNanoseconds(2020, 1, 15, 13, 0, 59, 123645, tzinfo=<UTC>),
   'id': 'TyRIPUKXWjRzvWIoTTQR'},
  '_exists': True,
  'read_time': seconds: 1579118320
  nanos: 255319000,
  'create_time': seconds: 1579089659
  nanos: 330504000,
  'update_time': seconds: 1579089659
  nanos: 330504000}
]

Without order_by:
```python
>> [x.__dict__ for x in raw_from_firebase]
>> [
{'_reference': <google.cloud.firestore_v1.document.DocumentReference at 0x10fb23940>,
  '_data': {'name': 'AAAA',
   'date': DatetimeWithNanoseconds(2020, 1, 15, 13, 0, 59, 123645, tzinfo=<UTC>),
   'id': 'TyRIPUKXWjRzvWIoTTQR'},
  '_exists': True,
  'read_time': seconds: 1579118374
  nanos: 66381000,
  'create_time': seconds: 1579089659
  nanos: 330504000,
  'update_time': seconds: 1579089659
  nanos: 330504000},
 {'_reference': <google.cloud.firestore_v1.document.DocumentReference at 0x10fb239b0>,
  '_data': {'name': 'AAAA', 'id': 'gBuHk5abGnZWH2AJKkBJ'},
  '_exists': True,
  'read_time': seconds: 1579118374
  nanos: 66381000,
  'create_time': seconds: 1579089565
  nanos: 375880000,
  'update_time': seconds: 1579089565
  nanos: 375880000}
]

You can repair that:

>>  for obj in Name.objects.all():
          if obj.date is None:
             obj.date = None
             obj.save(update_fields=['date'])
>> list((x.id, x.date) for x in Name.objects.all().order_by('date'))
>> [('gBuHk5abGnZWH2AJKkBJ', None),
 ('TyRIPUKXWjRzvWIoTTQR',
  datetime.datetime(2020, 1, 15, 13, 0, 59, 123645, tzinfo=datetime.timezone(datetime.timedelta(0), '+00:00')))]

[jwitos]

Thanks @gameboy86 for looking into it, appreciate it very much. I really like the solution to quickly update all documents.
Thanks again & closing this issue