Index gets into bugged state when `documentDeletion` task fails due to transaction size (`MDB_TXN_FULL`) about meilisearch

[queer]

@Kerollmops:

• I previously ran 1.6.0 with ~550k documents, and upgraded to 1.7.6 ahead of the increased index size. I did rebuild my index entirely when moving to 1.7.6.
• apt-get update && apt-get upgrade
• Linux meili-prod 5.15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux, no Docker.

[queer]

I have not been able to reproduce this thus far and will close this unless it comes up again (:

[queer]

Hello 🌈 This has not occurred again since the first time it happened, so I'm unsure if it was "just" a transient issue. I'll be sure to update if it comes up again!

[queer]

To follow up on this, it doesn’t happen every day; today’s run went fine. It’s just concerning that it can happen at all and leave my index in a state where I have to fully delete and recreate it.

[Kerollmops]

Hey @queer 👋

Thank you very much for the report. It seems related to the deletion by filter, and it is not the first report we got about it. I'll create an issue to link them together and investigate this bug simultaneously. It seems to be the first time that this documentDeletion by filter triggers an MDB_TXN_FULL and corrupts the facet database...

However, I have a few questions:

• Have you indexed your 2.2 million documents on a v1.7.6 from the beginning, or did you upgrade from a previous version?
• How did you upgrade from the previous version if that was the case?
• What is the OS and architecture of your machine, are you running on Docker? uname -a

Have a good day 🌻