llama_bootstrap: failed to load model from '/model.bin' about localai

[mudler]

documentation is getting some revamp, and @mkellerman worked a nice integration with chatgpt-web: an e2e docker-compose file would be just great!

[mudler]

models are not bundled in the image due to licensing - models like gpt4all, alpaca, and vicuna are based on LLaMA from Facebook, which prohibits modifications, alterations, and re-distributions of the weights in every form. See for instance nomic-ai/gpt4all#75.

Sadly, until there is a model with a free license that allows re-distribution, we can't embed it in the image, or we risk yet another DCMA takedown. You need to get the model somehow, and specify it as described in https://github.com/go-skynet/llama-cli#using-other-models

[regstuff]

models are not bundled in the image due to licensing - models like gpt4all, alpaca, and vicuna are based on LLaMA from Facebook, which prohibits modifications, alterations, and re-distributions of the weights in every form. See for instance nomic-ai/gpt4all#75.

Sadly, until there is a model with a free license that allows re-distribution, we can't embed it in the image, or we risk yet another DCMA takedown. You need to get the model somehow, and specify it as described in https://github.com/go-skynet/llama-cli#using-other-models

I get this error despite mounting. Here's my command: sudo docker run -v ~/llama_models/gpt4-x-alpaca-13b-native-4bit-128g/gpt4-x-alpaca-13b-ggml-q4_1-from-gptq-4bit-128g/:/models -p 8080:14004 -ti --rm quay.io/go-skynet/llama-cli:v0.4 api --context-size 700 --threads 12 --alpaca true --model /models/model.bin
I have a model.bin file inside the ~/llama_models/gpt4-x-alpaca-13b-native-4bit-128g/gpt4-x-alpaca-13b-ggml-q4_1-from-gptq-4bit-128g folder

[mudler]

Can you try by using the MODEL_PATH env var instead?

sudo docker run -e MODEL_PATH=/models/model.bin -v ~/llama_models/gpt4-x-alpaca-13b-native-4bit-128g/gpt4-x-alpaca-13b-ggml-q4_1-from-gptq-4bit-128g/:/models -p 8080:14004 -ti --rm quay.io/go-skynet/llama-cli:v0.4 api --context-size 700 --threads 12 --alpaca true

Just noticed this is being set on the main container image, a fix is landing in master! (bf85a31)

[mudler]

@regstuff now the master image is fixed, you can also try with the same command but using quay.io/go-skynet/llama-cli:latest instead

[jonit-dev]

@mudler Its not working for me either....

➜  llama-cli docker run -ti --rm quay.io/go-skynet/llama-cli:v0.4  --instruction "What's an alpaca?" --topk 10000 --gpt4all=true --model ./models/ggml-alpaca-7b-q4.bin
llama_model_load: failed to open './models/ggml-alpaca-7b-q4.bin'
llama_bootstrap: failed to load model from './models/ggml-alpaca-7b-q4.bin'
Loading the model failed: failed loading model
➜  llama-cli 
➜  llama-cli docker run -ti --rm quay.io/go-skynet/llama-cli:v0.4  --instruction "What's an alpaca?" --topk  --model ./models/ggml-alpaca-7b-q4.bin         
➜  llama-cli docker run -ti --rm quay.io/go-skynet/llama-cli:v0.4  --instruction "What's an alpaca?" --topk 10000 --alpaca true --model ./models/ggml-alpaca-7b-q4.bin
llama_model_load: failed to open '/model.bin'
llama_bootstrap: failed to load model from '/model.bin'
Loading the model failed: failed loading model
➜  llama-cli docker run -ti --rm quay.io/go-skynet/llama-cli:v0.4  --instruction "What's an alpaca?" --topk 10000 --alpaca "true" --model ./models/ggml-alpaca-7b-q4.bin
llama_model_load: failed to open '/model.bin'
llama_bootstrap: failed to load model from '/model.bin'
Loading the model failed: failed loading model
➜  llama-cli docker run -ti --rm quay.io/go-skynet/llama-cli:v0.4  --instruction "What's an alpaca?" --topk 10000 --model ./models/ggml-alpaca-7b-q4.bin      
llama_model_load: failed to open './models/ggml-alpaca-7b-q4.bin'
llama_bootstrap: failed to load model from './models/ggml-alpaca-7b-q4.bin'
Loading the model failed: failed loading model
➜  llama-cli docker run -ti --rm quay.io/go-skynet/llama-cli:v0.4  --instruction "What's an alpaca?" --topk 10000 --model ./models/ggml-alpaca-7b-q4.bin --alpaca true
llama_model_load: failed to open './models/ggml-alpaca-7b-q4.bin'
llama_bootstrap: failed to load model from './models/ggml-alpaca-7b-q4.bin'
Loading the model failed: failed loading model
➜  llama-cli docker run -ti -e MODEL_PATH=/models/ggml-alpaca-7b-q4.bin --rm quay.io/go-skynet/llama-cli:v0.4  --instruction "What's an alpaca?" --topk 10000 --model ./models/ggml-alpaca-7b-q4.bin --alpaca true
llama_model_load: failed to open './models/ggml-alpaca-7b-q4.bin'
llama_bootstrap: failed to load model from './models/ggml-alpaca-7b-q4.bin'
Loading the model failed: failed loading model
➜  llama-cli docker run -ti -e MODEL_PATH=/models/ggml-alpaca-7b-q4.bin --rm quay.io/go-skynet/llama-cli:v0.4  --instruction "What's an alpaca?" --topk 10000                                                     
llama_model_load: failed to open '/models/ggml-alpaca-7b-q4.bin'
llama_bootstrap: failed to load model from '/models/ggml-alpaca-7b-q4.bin'
Loading the model failed: failed loading model
➜  llama-cli docker run -ti -e MODEL_PATH=/models/ggml-alpaca-7b-q4.bin --rm quay.io/go-skynet/llama-cli:v0.4  --instruction "What's an alpaca?" --topk 10000
➜  llama-cli docker run -ti --rm quay.io/go-skynet/llama-cli:latest  --instruction "What's an alpaca?" --topk 10000 --model ./models/ggml-alpaca-7b-q4.bin --alpaca true                                      
Unable to find image 'quay.io/go-skynet/llama-cli:latest' locally
latest: Pulling from go-skynet/llama-cli
3e440a704568: Already exists 
68a71c865a2c: Already exists 
670730c27c2e: Already exists 
5a7a2c95f0f8: Already exists 
db119aaf144b: Already exists 
92ac76a462cb: Pull complete 
5997e4205ef7: Pull complete 
33d4a96cf7d6: Pull complete 
c8a35e5c3705: Pull complete 
abacb88fc6dd: Pull complete 
756caf9df70c: Pull complete 
0a7f01cc46c5: Pull complete 
92ed784c8873: Pull complete 
Digest: sha256:3698dea8ece687b23903afe347cee47b37d6883053533eacfab26619b55b97c7
Status: Downloaded newer image for quay.io/go-skynet/llama-cli:latest
llama_model_load: failed to open './models/ggml-alpaca-7b-q4.bin'
llama_bootstrap: failed to load model from './models/ggml-alpaca-7b-q4.bin'
Loading the model failed: failed loading model
➜  llama-cli docker run -ti --rm quay.io/go-skynet/llama-cli:latest  --instruction "What's an alpaca?" --topk 10000 --alpaca rue --model ./models/ggml-alpaca-7b-q4.bin
llama_model_load: failed to open ''
llama_bootstrap: failed to load model from ''
Loading the model failed: failed loading model
➜  llama-cli 
➜  llama-cli docker run -ti --rm quay.io/go-skynet/llama-cli:latest  --instruction "What's an alpaca?" --topk 10000 --alpaca true --model ./models/ggml-alpaca-7b-q4.bin
llama_model_load: failed to open ''
llama_bootstrap: failed to load model from ''
Loading the model failed: failed loading model
➜  llama-cli docker run -ti --rm quay.io/go-skynet/llama-cli:latest  --alpaca true --instruction "What's an alpaca?" --topk 10000 --model ./models/ggml-alpaca-7b-q4.bin              
llama_model_load: failed to open ''
llama_bootstrap: failed to load model from ''
Loading the model failed: failed loading model
➜  llama-cli docker run -ti --rm quay.io/go-skynet/llama-cli:latest  --instruction "What's an alpaca?" --topk 10000 --model ./models/ggml-alpaca-7b-q4.bin --alpaca true              
llama_model_load: failed to open './models/ggml-alpaca-7b-q4.bin'
llama_bootstrap: failed to load model from './models/ggml-alpaca-7b-q4.bin'
Loading the model failed: failed loading model
➜  llama-cli docker run -ti --rm quay.io/go-skynet/llama-cli:latest  --instruction "What's an alpaca?" --topk 10000 --model ./models/ggml-alpaca-7b-q4.bin              
llama_model_load: failed to open './models/ggml-alpaca-7b-q4.bin'
llama_bootstrap: failed to load model from './models/ggml-alpaca-7b-q4.bin'
Loading the model failed: failed loading model

even with the latest image

[jonit-dev]

The project is great, but I'd recommend refactoring the documentation to make it clearer.

It's kind of confusing to understand what to do.

I'm also preparing a docker-compose.yml file which I can share when its done

[mudler]

Hi @jonit-dev ,

You need to specify a volume to docker so it mounts a path local to the host inside the container with -v, see the instructions here
https://github.com/go-skynet/llama-cli#using-other-models

For a docker compose file, have a look at #10

On the other hand I do completely agree, I will rework the documentation as soon as possible, there are many lacunas and also other new features being added that needs to be documented too.

[mudler]

instructions updated to run with docker-compose, and multi-model support too: https://github.com/go-skynet/llama-cli#usage

I'd close this issue for now, if you are still facing issues, just re-open it!