Getting model_path keyError about casalioy

[madeepakkumar1]

[hippalectryon-0]

Please include:

• your exact .env file

note: per the readme, python casalioy/ingest.py # optional <path_to_your_data_directory> -> the argument is the data directory, not a file

[madeepakkumar1]

Downloaded the model and kept those in models/ folder

Generic

MODEL_N_CTX=1024
TEXT_EMBEDDINGS_MODEL=models/ggml-model-q4_0.bin
TEXT_EMBEDDINGS_MODEL_TYPE=LlamaCpp # LlamaCpp or HF
USE_MLOCK=true

Ingestion

PERSIST_DIRECTORY=db
DOCUMENTS_DIRECTORY=source_documents
INGEST_CHUNK_SIZE=500
INGEST_CHUNK_OVERLAP=50

Generation

MODEL_TYPE=LlamaCpp # GPT4All or LlamaCpp
MODEL_PATH=models/ggml-gpt4all-j-v1.3-groovy.bin
MODEL_TEMP=0.8
MODEL_STOP=[STOP]
CHAIN_TYPE=stuff
N_RETRIEVE_DOCUMENTS=100 # How many documents to retrieve from the db
N_FORWARD_DOCUMENTS=6 # How many documents to forward to the LLM, chosen among those retrieved
N_GPU_LAYERS=4

[hippalectryon-0]

First issue:

• ggml-model-q4_0.bin is a q4 model, which is deprecated. Use a q5 model, as written in the readme.

Are you on the latest master version ? If not, which one ?

[madeepakkumar1]

using latest main branch, do i need to switch ?

[hippalectryon-0]

Nope.

Can you replace your .env by the exact content of example.env and report the result here ?

[madeepakkumar1]

Thanks, it worked with default settings, But error in startLLm.py

$ python casalioy/startLLM.py
found local model at models/sentence-transformers/all-MiniLM-L6-v2
found local model at models/eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin
llama.cpp: loading model from models/eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin
terminate called after throwing an instance of 'std::runtime_error'
what(): read error: Is a directory
Aborted (core dumped)

[madeepakkumar1]

Output for ingest.py

$ python casalioy/ingest.py source_documents/AAR_20211231_CA1363851017_AR.pdf y
found local model at models/sentence-transformers/all-MiniLM-L6-v2
found local model at models/eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin
Deleting db...
Scanning files
0.0% [=================================================================================================================================>] 0/ ? eta [?:??:??]
Done

[hippalectryon-0]

python casalioy/ingest.py source_documents/AAR_20211231_CA1363851017_AR.pdf y

That's a wrong command. You need to provide a directory, not a file:

python casalioy/ingest.py source_documents/

[madeepakkumar1]

$ python casalioy/ingest.py source_documents/
found local model at models/sentence-transformers/all-MiniLM-L6-v2
found local model at models/eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin
Scanning files
Processing AAR_20211231_CA1363851017_AR.pdf
Processing 1393 chunks
Creating a new collection, size=384
Saving 1000 chunks
Saved, the collection now holds 999 documents.
embedding chunk 1001/1393
Saving 393 chunks
Saved, the collection now holds 1381 documents.
Processed AAR_20211231_CA1363851017_AR.pdf
Processing aapl-20220924.txt
Processing 565 chunks
Saving 565 chunks
Saved, the collection now holds 1943 documents.
Processed aapl-20220924.txt
Processing sample.csv
Processing 9 chunks
Saving 9 chunks
Saved, the collection now holds 1952 documents.
Processed sample.csv
Processing shor.pdf
Processing 22 chunks
Saving 22 chunks
Saved, the collection now holds 1974 documents.
Processed shor.pdf
Processing state_of_the_union.txt
Processing 90 chunks
Saving 90 chunks
Saved, the collection now holds 2064 documents.
Processed state_of_the_union.txt
Processing LLAMA Leveraging Object-Oriented Programming for Designing a Logging Framework-compressed.pdf
Processing 14 chunks
Saving 14 chunks
Saved, the collection now holds 2078 documents.
Processed LLAMA Leveraging Object-Oriented Programming for Designing a Logging Framework-compressed.pdf
Processing Constantinople.docx
Processing 13 chunks
Saving 13 chunks
Saved, the collection now holds 2090 documents.
Processed Constantinople.docx
Processing Easy_recipes.epub
[nltk_data] Downloading package punkt to /home/ubuntu/nltk_data...============================================> ] 7/ 9 eta [00:12]
[nltk_data] Unzipping tokenizers/punkt.zip.
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data] /home/ubuntu/nltk_data...
[nltk_data] Unzipping taggers/averaged_perceptron_tagger.zip.
Processing 31 chunks
Saving 31 chunks
Saved, the collection now holds 2121 documents.
Processed Easy_recipes.epub
Processing Muscle Spasms Charley Horse MedlinePlus.html
Processing 15 chunks
Saving 15 chunks
Saved, the collection now holds 2136 documents.
100.0% [===================================================================================================================================>] 9/ 9 eta [00:00]
Processed Muscle Spasms Charley Horse MedlinePlus.html
Done

[hippalectryon-0]

That looks correct. What about startLLM.py now ?

[madeepakkumar1]

Same error

$ python casalioy/startLLM.py
found local model at models/sentence-transformers/all-MiniLM-L6-v2
found local model at models/eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin
llama.cpp: loading model from models/eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin
terminate called after throwing an instance of 'std::runtime_error'
what(): read error: Is a directory
Aborted (core dumped)

[madeepakkumar1]

My system has GPU
$ nvidia-smi
Wed May 17 09:29:11 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02 Driver Version: 530.30.02 CUDA Version: 12.1 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 On | 00000000:00:1E.0 Off | 0 |
| N/A 33C P8 9W / 70W| 2MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+

[hippalectryon-0]

This is most likely not GPU related.

Just to make sure, your .env is the exact same as example.env and you haven't modified any source file ? Because it sounds like you're forwarding a directory to the LLM at some point.
Running python casalioy/startLLM.py works out-of-the-box for me.

[madeepakkumar1]

Yes, using exactly same example.env post renaming it to .env

Generic

MODEL_N_CTX=1024
TEXT_EMBEDDINGS_MODEL=sentence-transformers/all-MiniLM-L6-v2
TEXT_EMBEDDINGS_MODEL_TYPE=HF # LlamaCpp or HF
USE_MLOCK=true

Ingestion

PERSIST_DIRECTORY=db
DOCUMENTS_DIRECTORY=source_documents
INGEST_CHUNK_SIZE=500
INGEST_CHUNK_OVERLAP=50

Generation

MODEL_TYPE=LlamaCpp # GPT4All or LlamaCpp
MODEL_PATH=eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin
MODEL_TEMP=0.8
MODEL_STOP=[STOP]
CHAIN_TYPE=stuff
N_RETRIEVE_DOCUMENTS=100 # How many documents to retrieve from the db
N_FORWARD_DOCUMENTS=6 # How many documents to forward to the LLM, chosen among those retrieved
N_GPU_LAYERS=4

[hippalectryon-0]

What's the output of md5sum models/eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin ?

[madeepakkumar1]

Do i need to replace model_path with this? How to get the output ?

[hippalectryon-0]

It's not a python command. Just run it in the terminal.

Expected output:

❯ md5sum models/eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin
29e959c57faa0bcdf95b1ba5f7c9e968  models/eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin

[madeepakkumar1]

Thanks, getting below
$ md5sum models/eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin
md5sum: models/eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin: Is a directory

[hippalectryon-0]

aaah ok the problem is a lot easier then x)
Hotfix for you: in .env replace

MODEL_PATH=eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin

by

MODEL_PATH=eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin/ggml-vic7b-q5_1.bin

[su77ungr]

YOU HAVE ACCESS TO A TESLA GPU? @madeepakkumar1? Is this your own setup?

[madeepakkumar1]

Thanks @hippalectryon-0 worked after giving full path of the model
Sorry @su77ungr I don't have Tesla GPU setup, but i have Nvidia GPU setup, let me know if I could help here

[su77ungr]

@madeepakkumar1 nvidia-smi outputed Nvidia Tesla T4 GPU so this should be you GPU haha. Nice card. Ignore this message.