RAG - data cleaning before adding Knowledge base document about phidata

[ashpreetbedi]

I completely agree @mathieuchateau

A current fix would be to load the Documents manually instead of using the KnowledgeBase, something like:

reader = PDFReader()
pdf_documents: List[Document] = reader.read(uploaded_file)
for doc in pdf_documents:
    doc.content = clean_content(doc.content)

assistant.knowledge_base.load_documents(pdf_documents, upsert=True)

You could also do it with a Website reader:

scraper = WebsiteReader(max_links=2, max_depth=1)
web_documents: List[Document] = scraper.read(input_url)
for doc in web_documents:
    doc.content = clean_content(doc.content)

assistant.knowledge_base.load_documents(web_documents, upsert=True)

Example code

A long term approach would be to expose this as a parameter in the KnowledgeBase object, that accepts a "pre_process()" function, which runs on the content to clean it.

What do you think?

[mathieuchateau]

Looks promising, i will dig. I already have some functions to clean text like from tweets etc. I am mass injecting 5000 documents with 80 pages each in average. Maybe it’s too ambitious ? I bumped up limits on how many documents from knowledge base can be used regards, Mathieu Chateau Le mar. 23 avr. 2024 à 18:47, Ashpreet ***@***.***> a écrit :

…

I completely agree @mathieuchateau <https://github.com/mathieuchateau> A current fix would be to load the Documents manually instead of using the KnowledgeBase, something like: reader = PDFReader() pdf_documents: List[Document] = reader.read(uploaded_file) for doc in pdf_documents: doc.content = clean_content(doc.content) assistant.knowledge_base.load_documents(pdf_documents, upsert=True) You could also do it with a Website reader: scraper = WebsiteReader(max_links=2, max_depth=1) web_documents: List[Document] = scraper.read(input_url) for doc in web_documents: doc.content = clean_content(doc.content) assistant.knowledge_base.load_documents(web_documents, upsert=True) Example code <https://github.com/phidatahq/phidata/blob/main/cookbook/llms/groq/rag/app.py#L138-L141> A long term approach would be to expose this as a parameter in the KnowledgeBase object, that accepts a "pre_process()" function, which runs on the content to clean it. What do you think? — Reply to this email directly, view it on GitHub <#182 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCFMOKAJ3SHG7CAB2VMRODY62GDPAVCNFSM6AAAAABGVHT7OKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZSHEYDOOBVGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>