Out of memory when populating large dataset about django-elasticsearch-dsl

[lowdeyou]

I experienced the same issue with MySQL database. Are you also using MySQL database for Django?

I have done some testing and attempted to replicate this issue on both MySQL and Postgres databases, and I found that Postgres does not encounter Out Of Memory issues when dealing with large amount of data. Have not tested with other databases.

I did some investigation and found that the issue stems from Django iterator on this line:

django-elasticsearch-dsl/django_elasticsearch_dsl/documents.py

Line 105 in d29d771

return qs.iterator(**kwargs)

It seems that for MySQL, the iterator will still load the entire data set regardless, ignoring the chunk_size argument.
Not too sure why the difference in behavior but I suspect could be due this:
https://docs.djangoproject.com/en/4.1/ref/models/querysets/#without-server-side-cursors

To get around this issue, I wrote a custom iterator that does the chunking on the app-level, which should behave correctly regardless of which database you use:
lowdeyou@b43e6d2