API : Output is cut about localai

[mudler]

this hasn't really anything to do with the API itself, but rather with the model/llama: try setting an higher context when starting the API and most importantly a higher temperature. also, I've seen better answer with topk set to higher values (e.g. 10000).

[daedric7]

Ok, new example:

curl --location --request POST 'http://10.0.1.11:8080/predict' --header 'Content-Type: application/json' --data-raw '{
    "text": "What is the distance between the sun and the nearest star?",
    "topP": 0.8,
    "topK": 10000,
    "temperature": 1.1,
    "tokens": 100
}'

Output:

{"prediction":"\nThe distance between the sun and the nearest star is about 4.3 light years. This is the approximate distance between the stars Proxima Centauri and Alpha Centauri, which are located in the constellation Centaurus.\nHow far away is the nearest star?\nWhat is the closest star to Earth?\nWhat is the closest star to our solar system?\nWhat is the closest star to the sun?\nWhat is the closest star to our galaxy?\nWhat is the closest star to our solar system?\nWhat is the closest star to Earth?\nWhat is the closest star"}

Can you clarify what you mean with "higher context when starting the API" ?

[mudler]

Two issues I see here:

1. the temperature needs to be between 0 and 1
2. The API doesn't inject a template for talking to the instance, while the CLI does. You have to use a prompt similar to what's described in the standford-alpaca docs: https://github.com/tatsu-lab/stanford_alpaca#data-release

[mudler]

Closing, as doesn't seem related directly to llama-cli