Completion endpoint returns same response repeatedly about localai

[holzmichlnator]

I noticed this behaviour too while working with LocalAI. I also tried to set the prompt_cache_all option to false but this didn't change anything. Setting temperature, top_k, top_p, seed to different values also doesn't change anything. The strangest part is, it keeps generating the same output even after recreating the whole container except the ID is now different.
I also tried to modify the prompt slightly by adding an "!" at the end for example, but even then, the answer is identical.

I'm using localai/localai:v2.9.0-cublas-cuda12

[mudler]

While i'm having a look at this - the options seems passed just fine up to the gRPC server and llama.cpp.

However, even when I set slot.params.seed = time(NULL); right before we configure the slot ready to be processed

LocalAI/backend/cpp/llama/grpc-server.cpp

Line 879 in dc919e0

llama_set_rng_seed(ctx, slot->params.seed);

( as llama.cpp does as well ), it seems completely ignored.

Also to note, that's the full json data printed out (and looks indeed setting seed, top_k, and top_p accordingly):

12:36PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:40341): stdout {"cache_prompt":false,"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"mirostat":0,"mirostat_eta":0.0,"mirostat_tau":0.0,"n_keep":0,"n_predict":-1,"penalize_nl":false,"presence_penalty":0.0,"prompt":"Instruct: tell me a story about llamas\nOutput:\n","repeat_last_n":0,"repeat_penalty":0.0,"seed":-1,"stop":[],"stream":false,"temperature":0.20000000298023224,"tfs_z":0.0,"top_k":40,"top_p":0.949999988079071,"typical_p":0.0}

so it looks definitely something is off in the llama.cpp implementation, as our is just a gRPC wrapper on top of the http example (with few edits to avoid bugs like #1333 )

[mudler]

Ok, tracing seed seemed to be just a red herring and dragged me in the wrong direction. I'm tracing back the usage and it looks like it is because it doesn't have enough candidates to select the tokens.

I've tried to switch sampler with phi-2 and I got finally a more undeterministic result. It looks to me it is very much depending on the model/sampler strategy - mirostat can get more candidates, while the temperature sampler has less to select which (so it is more deterministic).

E.g. with phi-2:

name: phi-2
context_size: 2048
f16: true
gpu_layers: 90
mmap: true
trimsuffix:
- "\n"
parameters:
  model: huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
  temperature: 1.0
  top_k: 40
  top_p: 0.95
mirostat: 2
mirostat_eta: 1.0
mirostat_tau: 1.0

  seed: -1
template:
  chat: &template |
    Instruct: {{.Input}}
    Output:
  completion: *template

usage: |
      To use this model, interact with the API (in another terminal) with curl for instance:
      curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
          "model": "phi-2",
          "messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
      }'

[mudler]

There were incongruences with the docs, updated also the samples in #1820. If the issue persist feel free to re-open