ERROR: stderr update_slots : failed to find free space in the KV cache about localai

[mudler]

@netandreus this seems to likely happen when the prompt exhausts the context size - can you check if that's causing issues in your case by bumping the context size?

In any case sounds reasonable to bail out early instead of trying to free space in the KV cache. this seem also related to #2258 - can you also try by setting batch to 1 in the model configuration and see if keeps happening?

parameters:
  batch: 1

[DavidGOrtega]

related to #2258

[netandreus]

Thank you for assistance, I will check.

[imihic]

It seems that this error also happens if we enable parallel llama.cpp processing. For an example, setting the context size to 8192 and the number of parallel processes to 20, the token stream generation always stops at around 410 characters, which is roughly equal to 8192 divided by 20.

So, instead of each process allocating 8192 context window size, as specified in the .env/yaml file, the backend takes this value and splits it between all the processes.

Is this a bug or expected behaviour? If it's expected it might not be a bad idea to clarify this behaviour in the documentation.