hyperledger-labs / aifaq

AI FAQ Proof-of-Concept project: it provides a chatbot that replies to the questions on Hyperledger Ecosystem

Hyperledger QA PoC version 2

The scope of this Hyperledger Labs project is to support the users (users, developer, etc.) to their work, avoiding to wade through oceans of documents to find information they are looking for. We are implementing an open source conversational AI tool which replies to the questions related to specific context. This is a proof-of-concept software which allows to create a chatbot using Google Colab (or local notebook which requires GPU). Here the official Wiki page: Hyperledger Labs aifaq. Please, read also the Antitrust Policy and the Code of Conduct. The meeting invitation is on the Hyperledger Labs calendar: [Hyperledger Labs] FAQ AI Lab calls.

Background

The system is an open source Jupyter Notebook (derived from here medium.com) which implements an AI chatbot. The idea is to implement an open source framework/template, as example, for other communities. Last results in open LLMs allow to have good performance using common HW resources.
Below the application architecture:

We use RAG (Retrieval Augmented Generation arxiv.org) for question answering use case. That technique aims to improve LLM answers by incorporating knowledge from external database (e.g. vector database).

The image depicts two workflow:

1. The data ingestion workflow
2. The chat workflow

During the ingestion phase, the system loads context documents and creates a vector database. In our case, the document sources are:

• An online software guide (readthedocs template)
• The GitHub issues and pull requests

After the first phase, the system is ready to reply to user questions.

Currently, we use the open source HuggingFace Zephyr-7b-alpha. But, in the future we want to investigate other open source models. Moreover, the User Interface uses Gradio.

Open Source Version

The software is under Apache 2.0 License (please check LICENSE and NOTICE files included). For the dependencies, it is ASF 3rd Party License Policy compliant: the LICENSE file contains "pointers" to the dependency's licenses and a list of Apache 2.0-licensed dependecies (Assembling LICENSE and NOTICE files).

Installation

Below the main steps to set up the system:

1. Download the hyperledger_aifaq_poc_v3.ipynb notebook file from the src folder
2. Create a new Google Colab notebook
3. Load the downloaded notebook file
4. Set up the runtime GPU
5. Copy the ReadTheDocs on Google Drive
6. Set up the Google Drive mount path
7. Set the URL and GitHub repo document sources
8. Create a new GitHub personal token
9. Add the token, as new secret, to the Google Colab notebook

The first step is straightforward: just click the src folder to open it, then click the hyperledger_aifaq_poc_v3.ipynb file and the click the button below:

Now, in Google Drive click on New button -> Other and Google Colaboratory

Inside the new notebook, select the File menu, then select Load notebook and then click on the "Browse" button and select the downloaded file (hyperledger_aifaq_poc_v3.ipynb).

We need a GPU to execute the notebook. So, we can set it up from the Runtime menu, then change runtime:

If you have a free account you can use only the T4 GPU.

Now, download the on-line ReadTheDocs documentation, e.g. from Hyperledger fabric ReadTheDocs:

Click on the v: realease-2.5 and then click on Download:

It should open a web page where it is possible to download the HTML version:

Unzip the file and copy folders and files on your Google Drive:

Now, we can set the document sources. The notebook takes the documents for RAG from three sources:

1. An online website
2. A local ReadTheDocs documentation
3. A GitHub repository

The image below shows how to set them up:

In our case, we get the Hyperledger fabric wiki page and its GitHub repository (getting issues and pull requests). Into url string we specify the website, while in repo string we set the GitHub repository.\

While, we should specify the ReadTheDocs folderpath and use it below.

From your personal GitHub account, inside the profile settings, select the developer settings:

Then select the fine-grained token

and click on the generate button: now copy the token.\ Into the Google Colab notebook, select the secret key and add a new secret, like the image below:

• The token must have the access to the notebook
• The name should be GITHUB_PERSONAL_ACCESS_TOKEN
• Past it inside the Value field

Usage

Now, we can test the PoC by executing the notebook: in Google Colab notebook -> Runtime menu, select Execute all:

• It will take 5-15 minutes (it depends on the GPU and the documents)
• When the execution finishes, it loads an UI which allows to ask questions and replies in around 30 seconds

Below an example:

For any reason, please, contact us on Discord Channel:

• Server: Hyperledger
• Channel: #aifaq

Current version notes

That is a proof-of-concept: a list of future improvement below:

1. We want to implement a prototype starting from that PoC: container architecture installed on a GPU Cloud Server
2. At the same time, we'd like to pass to the next step: the Hyperledger Incubation Stage
3. We will investigate other open source models
4. Evaluation of the system using standard metrics
5. We would like to improve the system, some ideas are: fine-tuning, Corrective RAG, Decomposed LoRA
6. Add "guardrails" which are a specific ways of controlling the output of a LLM, such as talking avoid specific topics, responding in a particular way to specific user requests, etc.

aifaq's Issues

LLM Infrence speed is slow want to improve it by Groq

• I noticed that the after asking it usually give answer very approx 2min so I was thinking to play around other LLM Infrence Model like Groq to make the output real-time.

• Does this project active and how I can contribute to this with other open source models

• I have setup the project on Google colab
  

It would be great if you can guide me in the process @gcapuzzi

Suggestion to implement latest models like llama-3 or Mistral 7B and should be deployed on a proper FrontEnd Project

I suggest that we use llama models or Mistral 7B foundation models to develop a comprehensive dashboard. We can implement it using React and then deploy it on a cloud server. This approach will ensure scalability and proper accessibility.

@gcapuzzi

Feature : Add CONTRIBUTING.md and requirements.txt files

It would be better to move correct requirements to a requirements.txt file with following content:

transformers
torch
gradio
bs4
safetensors
accelerate
sentencepiece
bitsandbytes
sentence-transformers
chromadb
langchain

I'd also like to create a CONTRIBUTING.md file describing how to setup the environment.

Any other recommendations @gcapuzzi ?