csabaconsulting / thruthinkcohereweaviatechat

Cohere and Weaviate powered ThruThink support chat on Streamlit

ThruThink® Support Chat Agent utilizing RAG Fusion, powered by Cohere & Weaviate

• 5 minute Demo Pitch (at the point where the Stremlit app is shown)
• Slide deck PDF (pitched before the Streamlit app) (pptx)
• The project's hackathon page on lablab.ai

The main business goal is to develop a support chat agent for the investment projection web application ThruThink®.

• ThruThink® is a business budgeting on-line app to create professional budgets and forecasts
• A product of literally decades of experience and careful thought, and thousands of calculations
• Thru-hiking, or through-hiking, is the act of hiking an established long-distance trail end-to-end continuously
• There are no dedicated personnel for support chat agent roles, it had a “classic” chat agent integration in the past
• An LLM and RAG (Retrieval Augmented Generation) powered chat agent could be invaluable, given that
  1. It stays relatively grounded
  2. It Won’t hallucinate* wildly

Desired abilities:

• Main goal: answer ThruThink® software specific questions such as: "In ThruThink can I make adjustments on the Cash Flow Control page?"
• Nice to have: answer more generic questions such as: "How much inventory should I have?"

The bulk of the knowledge base consists of close to 190 help topics also divided into a few dozen categories. That's more than nothing, however users can ask such a wide variety of questions that chunking these documents may not provide a nice ground for a good vector match in the embedding space. To increase the performance of the chat agent I employ several techniques.

Achievements:

1. With a synthetic data generation I enriched the knowledge base. I coined this QBRAG (QnA Boosted RAG) because I'm using the same QnA data I already generated and curated for potential fine tuning purposes. The same dataset can be used to enrich the vector indexed knowledge as well.
2. The highlight of my submission is RAG Fusion (see article).
3. I utilize Weaviate for vector storage, embedding, matching and retrieval. I use Cohere for multiple language models: fine tuned chat model and also co.chat with a web connector advanced feature in different stages of the chain.
4. I perform metadata helped retrieval since I ingest and index the help documents' titles and categories.
5. I also use LangChain to construct some stages of the chain.
6. The front-end is powered by and hosted on Streamlit, I highly customized the view which also features linked references.
7. After the fusion and re-ranking I provide the user with both results from a more traditional RAG grounded co.chat call and also a web-connector powered other call (that is also augmented to provide guidance) to show both information so the user can get the best of both worlds.
8. Since I have to control several stages of the chain for the fusion, I was not able to use such high level LangChain constructs as ConversationalRetrievalChain or RetrievalQA, so co.chat's ability to handle the conversation for me (via conversation_id) made my job much easier than I'd have to work for history / memory functionality and other building blocks.

RAG Fusion:

1. Since users might ask questions which don't match well into the QnA questions in its particular form, but it is still covered by the knowledge base, the application first generates variations of the user's query with the help of a fine tuned Cohere model. The hope is that some of these variations may match closer to some QnA or help data chunks.
2. The document retrieval then happens for all of the query variations.
3. There's a reciprocal rank fusion which concludes a fused list of documents across all the variations.
4. We'll take the top k of those documents and perform final two RAG calls which supply the displayed data. Both RAG calls use the cutting edge co.chat, one of the calls is document based, and the other is a web connector based (but still document augmentation helped for better result).

Note that the application and all the help documents are English. Therefore we used the embed-english-v2.0 with cosine similarity. Knowing we need to perfrom only in English domain we can expect possibly slightly better performance. We need to pay attention to the similarity (multi lingual embedding uses dot product). Also refer to https://github.com/CsabaConsulting/Cohere/blob/main/WeaviateInit.ipynb.

Other achievements:

• I added the capability to the https://github.com/nestordemeure/question_extractor open source project to continue a long running interrupted QnA generation session (it can take many hours with default rate limits on both OpenAI or AnyScale). See my open repository https://github.com/CsabaConsulting/question_extractor
• I developed a script which can convert the standard format fine tuning jsonl into a set of markdown files (QnA grouped by help topics) for Weaviate indexing. The script breaks apart the questions.jsonl into the files based on which file the questions originally sourced from. This is needed for the QBRAG synthetic data indexing. For the conversion see the https://github.com/CsabaConsulting/question_extractor/blob/main/augment_prep.py script.
• I added capability for my script to produce a fine tuning file for the Cohere LLM model (see https://github.com/CsabaConsulting/question_extractor/blob/main/fine_tune_prep.py).

Kindly look at the development and experimentation IPython notebooks and scripts in the https://github.com/CsabaConsulting/Cohere repository. These were used to establish the Weaviate schema for ingestion / indexing / retrieval, and also testing retrieval and building up the parts for the RAG Fusion.

Future plans:

• Decrease runtime by running the variation document retrievals in parallel, this is a Streamlit specific tech challenge with asyncio / await.
• Decrease runtime by running the final two co.chat RAG calls in parallel, this is a Streamlit specific tech challenge with asyncio / await.
• Make the citation linking nicer and other UI enhancements.
• Measure how much the RAG Fusion (if at all) improves answer quality. Measure the trade-off factoring in extra latency and potential token usage increase which also means cost increase.
• Integrate the agent into ThruThink which uses ASP.NET MVC / C# / Azure technology stack, but not open source. In that final deployment I'll be able to open up referred help topics using the meta-data I get back as part of the query results document metadata.
• Add filter against harmful content, for example using Google PaLM2's safety attributes.

thruthinkcohereweaviatechat's Issues

Simplify implementation with LangChain's MultiQueryRetriever

See https://www.youtube.com/watch?v=VFf8XJUIHnU
https://python.langchain.com/docs/modules/data_connection/retrievers/MultiQueryRetriever
langchain-ai/langchain#9231

Adding buttons with example queries for an easier demo / test experience

Sebastian Witalec's review asks for "It would be great if you could provide a few examples on how best to play with your demo". I think providing some preset queries which could be easily invoked by a few clicks would satisfy a user who is feeling out the app.

Executing the variation document retrievals and the twin co.chat queries paralelly

Use asyncio in Streamlit, we can parallelize:

1. The document retrieval for the variations
2. The twin calls (document based and web connector) of co.chat

(The latter two are the real latency hogs)

Highlight document chunk section in the document reference side

Currently I only handle and interpolate the references between the return text to the referenced documents. However when the retrieved document is not a QnA (which is a wholesome chunk), but a chunk from a help document, I could get the whole document and I can highlight which chunk section of that document made it to the retrieval. I'd need to design the UX, because even right now the document reference section can be long. Maybe looking for a Streamlit accordion or something.

Prompt engineer the query variation prompt

Came across https://github.com/pinecone-io/examples/blob/master/learn/generation/langchain/handbook/10-langchain-multi-query.ipynb in #6 https://www.youtube.com/watch?v=VFf8XJUIHnU

His prompt:

template = """
Your task is to generate 3 different search queries that aim to
answer the user question from multiple perspectives. The user questions
are focused on Large Language Models, Machine Learning, and related
disciplines.
Each query MUST tackle the question from a different viewpoint, we
want to get a variety of RELEVANT search results.
Provide these alternative questions separated by newlines.
Original question: {question}
"""

Refactor code further

Somehow extract the lengthy initialization.
Use black or other Python formatting on the source code.

RAG with Llama Index and Weaviate

Explore: https://dstack.ai/examples/llama-index-weaviate/

Better error handling

For example if any clients or API endpoints could not be instantiated or similar corner cases

Use cohere re-ranker after we got the reranked list of the fused documents

This could easily be an extra step between the final RAG preparation and the fusion reranking.
See https://github.com/pinecone-io/examples/blob/master/learn/generation/better-rag/00-rerankers.ipynb

import cohere

os.environ["COHERE_API_KEY"] = os.getenv("COHERE_API_KEY") or getpass.getpass()
co = cohere.Client(os.environ["COHERE_API_KEY"])
rerank_docs = co.rerank(
    query=query, documents=docs.keys(), top_n=25, model="rerank-english-v2.0"
)

Study applicability of self query retrieval

Advised by Philip Vollet (Weaviate):

• safe search
• filtering
• self query retrieve
  https://medium.com/@kamaljp/deep-dive-into-self-query-retriever-in-langchain-exploring-pros-of-building-sq-retriever-with-cf12f3eed1f3

Follow-up questions trick the fusion

Example:

Q: "What do you know about ThruThink Deal Score?"
A: "ThruThink Deal Score is a proprietary feature1.1 that evaluates multiple factors1.1 involving historical information, the transaction itself and the projections1.1 which result in an overall grade or score for the "Deal"1.1.
It is an optional feature1.2, that is turned off by default1.2 within the ThruThink web application. It must be manually turned on by the user1.2 in the Workflow menu, Step 71.2, by clicking the button for ThruThink Evaluation inputs1.2 and choosing "Yes"1.2.
The ThruThink Deal Score provides an "A", "B", "C", "D" or "F"1.2 rating of the Deal with "A"1.2 being the highest rating. The rating evaluates the Deal or comparative benefit to the Owner as presented by the User Inputs.1.2
Would you like to know more about the ThruThink Evaluation?"
Q: "Yes I would like"

Currently we generate variations for the current question. The variation generator LLM doesn't have the context of the co.chat. We'll need to work on this.

Explore Weaviate's new Verba RAG platform

https://github.com/weaviate/Verba
This would be an alternative to Streamlit?