warning: failed to mlock 245760-byte buffer (after previously locking 0 bytes): Cannot allocate memory llm_load_tensors: mem required = 46494.72 MB (+ 1280.00 MB per state) about llama-api

[c0sogi]

By default, the mlock parameter defaults to True in this application, but False in oobabuga, which is why I think you're seeing different results.

The reason why mlock fails is probably because mlock is limited on your system. If you are on Linux, consider using the 'ulimit -l unlimited' option. or, pass 'use_mlock=False' to your LlamaCppModel. Actually, you shouldn't have any problems using it without mlock.

[Dougie777]

I already set ulimit to unlimited. I will try use_mlock=False. Thanks!

[Dougie777]

That worked. I no longer get the error. THANKS!

[delta-whiplash]

how can I use_mlock=False on docker setup please