bug: nitro attempts to load nonexistent "ggml-model-f16.gguf" model about nitro

[shavit]

Sorry, I got this error from server not nitro. The nitro server will start without a model argument, just make sure to use the absolute path to the file in your request.

[shavit]

@zeozeozeo this is the default model, but you can use other models with -m MODEL_FILE instead. The server will not start if the path is incorrect.

[zeozeozeo]

@zeozeozeo this is the default model, but you can use other models with -m MODEL_FILE instead. The server will not start if the path is incorrect.

hm, so to load new models I need to restart the nitro server each time?

[smathews]

I'm seeing the same issue. Regardless of what path I specify with /inferences/llamacpp/loadmodel it attempts to load the default model.

[CameronNg]

Hi @zeozeozeo, sorry for the late response.
Since your OS is Windows, the llama_model_path is a bit difference.

For example, here is my model path: "C:\Users\UserName\Downloads\nitro-win-amd64-avx2-cuda-11-7\llama-2-7b-model.gguf"
Then here is the correct request JSON to load model on Windows:

curl http://localhost:3928/inferences/llamacpp/loadmodel -d '{
  "llama_model_path": "C:\\Users\\UserName\\Downloads\\nitro-win-amd64-avx2-cuda-11-7\\llama-2-7b-model.gguf",
  "ctx_len": 512,
  "ngl": 100,
}'

FYI, the string models/7B/ggml-model-f16.gguf is the default model alias from llama.cpp.