memory allocation of 41/47 bytes failed about meilisearch

[dureuill]

Hello,

we found a fix, see #4707 for details

Considering the impact of this issue, we'll be releasing the fix in:

• a patch release for v1.8, and
• the next RC for v1.9

[dureuill]

Some progress in the investigation, thanks to the work of users in the Discord.

• it looks like vm.map_count increases without decrease
• the increase in maps is also accompanied by a memory increase that looks like a memory leak
• the error message alternate with an unrecoverable panic while setting the stack guard page
• confirmed to be a 1.8 regression

@irevoire has been trying to reproduce the issue on our end, more information about whether that attempt was successful to follow today.

[dureuill]

Hello @iamriajul, you can also downgrade to v1.7.6 which doesn't exhibit this behavior (v1.8 regression) but still has much better performance and a lot more features than v1.3.5

We're trying to understand the cause of the regression in v1.8. It looks like a memory leak indeed.

[dureuill]

Some news on this issue:

• @irevoire managed a reproduction of the leak on Meilisearch v1.8.0 and checked there was no reproduction in v1.7.6

v1.8.0, number of memory maps according to time

• The number of maps is linear in time.
• At the time the machine was stopped it had reached around 16k maps

v1.7.6, number of memory maps according to time

• The number of maps is constant in time
• it hovers around 400 maps

• @irevoire tested with a downgrade of charabia from v0.8.10 to v0.8.7 (the version it was in v1.7.6). It seems to fix the leak. We're now investigating the source of the memory leak in charabia

v1.8.0 with downgraded charabia, number of memory maps according to time

• The number of maps is constant in time
• it hovers around 400 maps

[dureuill]

⚠️ There was a setup issue in the previous test campaign; the leak is still present with charabia v0.8.7 and so was not introduced by charabia v0.8.10.

Continuing investigation...

[dureuill]

Next steps:

1. try and reproduce the issue to further debug with debug builds and hooks

[dureuill]

https://github.com/orgs/meilisearch/discussions/677#discussioncomment-9562242 could also be related

[iamriajul]

https://github.com/orgs/meilisearch/discussions/677#discussioncomment-9562242 could also be related

Thanks, I found this. Recently after a server migration, I migrated meilisearch to 1.8.1 from 1.3.5 and then nightmare started, I couldn't find the reason, then suspected docker resources limit issue (#4686) but after few days today meilisearch crashed again!!! So this saved me, by just downgrading to 1.3.5.

[dureuill]

Fixed in v1.8.3