Comments (6)
@imartinez I just realized with this issue that currently the ingest
file only loads a single file regardless of the number of documents in the source_documents
directory. The code changes the type of document loader for each file and then loads its content at the end (the very last file only).
# Load document and split in chunks
for root, dirs, files in os.walk("source_documents"):
for file in files:
if file.endswith(".txt"):
loader = TextLoader(os.path.join(root, file), encoding="utf8")
elif file.endswith(".pdf"):
loader = PDFMinerLoader(os.path.join(root, file))
elif file.endswith(".csv"):
loader = CSVLoader(os.path.join(root, file))
documents = loader.load() # loads only the last file content!
I am working on a fix right now.
from privategpt.
Not sure, maybe you can run a couple tests with fewer docs to check if that's the case. Thanks for your help!
from privategpt.
That's what I did. It works with 7-8 PDFs but then gives this error when I add more
from privategpt.
Ok great, thanks for sharing. Does it fail right away? It is interesting, will need to look into it. Please share your findings!
from privategpt.
It fails right at the beginning, see the output above. Let me know if you need anything else.
from privategpt.
i pulled the git today and yet get the same issue. I am not even uploading multiple files. i am just trying to check this using colab with state of the union.txt.
Loading documents from source_documents
Loaded 0 documents from source_documents
Split into 0 chunks of text (max. 500 tokens each)
llama.cpp: loading model from llm/ggml-model-q4_0.bin
llama.cpp: can't use mmap because tensors are not aligned; convert to new format to avoid this
llama_model_load_internal: format = 'ggml' (old version with low tokenizer quality and no mmap support)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 1000
llama_model_load_internal: n_embd = 4096
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 32
llama_model_load_internal: n_layer = 32
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 11008
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size = 4113748.20 KB
llama_model_load_internal: mem required = 5809.33 MB (+ 2052.00 MB per state)
...................................................................................................
.
llama_init_from_file: kv self size = 1000.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
Using embedded DuckDB with persistence: data will be stored in: db
Traceback (most recent call last):
File "/content/privateGPT/ingest.py", line 96, in
main()
File "/content/privateGPT/ingest.py", line 90, in main
db = Chroma.from_documents(texts, llama, persist_directory=persist_directory, client_settings=CHROMA_SETTINGS)
File "/usr/local/lib/python3.10/dist-packages/langchain/vectorstores/chroma.py", line 413, in from_documents
return cls.from_texts(
File "/usr/local/lib/python3.10/dist-packages/langchain/vectorstores/chroma.py", line 381, in from_texts
chroma_collection.add_texts(texts=texts, metadatas=metadatas, ids=ids)
File "/usr/local/lib/python3.10/dist-packages/langchain/vectorstores/chroma.py", line 159, in add_texts
self._collection.add(
File "/usr/local/lib/python3.10/dist-packages/chromadb/api/models/Collection.py", line 97, in add
ids, embeddings, metadatas, documents = self._validate_embedding_set(
File "/usr/local/lib/python3.10/dist-packages/chromadb/api/models/Collection.py", line 340, in _validate_embedding_set
ids = validate_ids(maybe_cast_one_to_many(ids))
File "/usr/local/lib/python3.10/dist-packages/chromadb/api/types.py", line 75, in maybe_cast_one_to_many
if isinstance(target[0], (int, float)):
IndexError: list index out of range
from privategpt.
Related Issues (20)
- How to use PGPT with LLM and Embeddings as OPENAI. HOT 1
- Chat via API HOT 1
- Issue while installation HOT 13
- [BUG] `make ingest` fails if path contains spaces HOT 1
- Increase Token
- Removing 'Additional Inputs' from UI HOT 1
- why did you remove the 'requirements.txt' from the repo? HOT 1
- [BUG] Another permissions error when installing with docker-compose HOT 2
- [QUESTION] Random docker instances are being created
- [QUESTION] Request, can we have docker installation documentation? HOT 3
- How can i contribute in project? HOT 1
- Dynamic database path HOT 1
- error in poetry install --extras "llms-llama-cpp vector-stores-qdrant ui embeddings-huggingface" HOT 2
- Recipes: Summarization
- [FEATURE] add chat history option
- Model won´t query when enabling Rerank in settings-local.yaml[BUG]
- [QUESTION] Support for Llama on Azure AI Studio HOT 3
- [QUESTION] Semantic Chunking HOT 1
- [BUG] UI keeps track of only the first 20 messages in the chat history
- [QUESTION] Is there a memory leak in huggingface embedding with pipeline mode HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from privategpt.