bmedi / chromadb-plugin-for-lm-studio

Plugin that creates a ChromaDB vector database to work with LM Studio while it's running in server mode!

🚀 Supercharge your LM Studio with a Vector Database!

Ask questions about your documents and get an answer from LM Studio!

Start a new "issue" if you want to request a feature or report a bug!

⚡GPU Acceleration⚡

GPU	Windows	Linux	Requirements
Nvidia	✅	✅	CUDA
AMD	⚠️ (see below)	✅	ROCm 5.6
Apple/Metal	✅

Prerequisites

‼️ Make sure to have at least 🐍Python 3.10 (haven't tested higher).
‼️ You must have both Git and git-lfs installed.
‼️ Processing .rtf files requires that you install Pandoc first.

Installation

‼️ For any commands that begin with python or pip, if you have both Python 2 and Python 3 installed, you should use Python3 or pip3 instead to make sure that the correct version of Python is used.

🪟WINDOWS INSTRUCTIONS🪟

Step 1 - Install GPU Acceleration Software

• 🟢 Nvidia GPU ➜ install CUDA 11.8

  Note that this installation is system-wide and it's not necessary to install within a virtual environment.

• 🔴 AMD GPU - Unfortunately, PyTorch does not currently support AMD GPUs on Windows (only Linux). However, it may be possible by using WSL within Windows pursuant to the instructions HERE and then access GPU-acceleration via HERE. This might also be helpful However, I do not have an AMD GPU so please let me know if you get it working with this method. If this does work for you, proceed to the instructions below on how to install my program within Linux.

Step 2 - Obtain Repository

• Download the latest "release" and unzip anywhere on your computer.

Step 3 - Virtual Environment

• Open the folder containing my repository files. Open a command prompt. Create a virtual environment:

python -m venv .

• Activate the virtual environment:

.\Scripts\activate

Step 4 - Upgrade pip

python -m pip install --upgrade pip

Step 5 - Install PyTorch

• 🟢 Nvidia GPUs:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

🔴 AMD GPU - Unfortunately, PyTorch does not currently support AMD GPUs on Windows (only Linux). However, it may be possible by using WSL within Windows pursuant to the instructions HERE and then access GPU-acceleration via HERE. However, I do not have an AMD GPU so please let me know if you get it working with this method. If this does work for you, proceed to the instructions below on how to install my program within Linux.

• 🔵 CPU only:

pip install torch torchvision torchaudio

Step 6 - Install Dependencies

pip install -r requirements.txt

Step 7 - Double check GPU-Acceleration

python check_gpu.py

🐧LINUX INSTRUCTIONS🐧

Step 1 - GPU Acceleration Software

• 🟢 Nvidia GPUs ➜ install CUDA 11.8

  Note that this installation is system-wide and it's not necessary to install within a virtual environment.

• 🔴 AMD GPUs ➜ install ROCm version 5.6 according to the instructions.

  Additionally, this repo might help, but I can't verify since I don't have an AMD GPU nor Linux.

Step 2 - Obtain Repository

• Download the latest "release" and unzip anywhere on your computer.

Step 3 - Virtual Environment

• Open the folder containing my repository files. Open a command prompt. Create a virtual environment:

python -m venv .

• Activate the virtual environment:

source bin/activate

Step 4 - Update Pip

python -m pip install --upgrade pip

Step 5 - Install PyTorch

• 🟢 Nvidia GPU:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

• 🔴 AMD GPU:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.6

• 🔵 CPU only:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

Step 6 - Install Dependencies

‼️On Linux systems run this first before installing the requirements.txt: sudo apt-get install portaudio19-dev and sudo apt-get install python3-dev

pip install -r requirements.txt

Step 7 - Double check GPU-acceleration

python check_gpu.py

🍎APPLE INSTRUCTIONS🍎

Step 1 - GPU Acceleration Software

• All Macs with MacOS 12.3+ come with 🔘 Metal/MPS support, which is the equivalent of CUDA and ROCm. However, you still need to install Xcode Command Line Tools.

Step 2 - Obtain Repository

• Download the ZIP file containing the latest release for my repository. Inside the ZIP file is a folder holding my repository. Unzip and place this folder anywhere you want on your computer.

Step 3 - Virtual Environment

• Open the folder containing my repository files. Open a command prompt. Create a virtual environment:

python -m venv .

• Activate the virtual environment:

source bin/activate

Step 4 - Update Pip

python -m pip install --upgrade pip

Step 5 - Install PyTorch

pip install torch torchvision torchaudio

Step 6 - Install Dependencies

‼️On MacOS systems run this first: brew install portaudio

pip install -r requirements.txt

Step 7 - Double check Metal/MPS-acceleration

python check_gpu.py

Transcription Instructions

As of release 2.1+, my program allows you to speak a question and have it transcribed to the system clipboard, which you can then paste into the LM Studio question box. It uses the "faster-whisper" library, which relies upon the powerful Ctranslate2 library and the state-of-the-art "Whisper" models.

🔥TRANSCRIPTION INSTRUCTIONS🔥

Compatibility Overview

⚡Transcription Acceleration⚡

	Acceleration Support	Requirements
Intel CPU	✅
AMD CPU	✅
Nvidia GPU	✅	CUDA
AMD GPU	❌	Will default to CPU
Apple CPU	✅
Apple Metal/MPS	❌	Will default to CPU

The type of acceleration you'll be using also determines the supported quantizations (discussed below).

Compatibility Checker

On Windows, simpy run ctranslate2_compatibility.exe.

On Linux or MacOS, follow the instructions HERE.

Changing Transcription Model or Quantization

All transcription settings can be changed in-program as of Version 2.5!

Usage

🔥USAGE INSTRUCTIONS🔥

Step 1 - Virtual Environment

Open a command prompt within my repository folder and activate the virtual environment:

NOTE: For 🍎Macs and 🐧Linux the command is: source bin/activate

.\Scripts\activate

Step 2 - Run Program

python gui.py

• ‼️ Only systems running Windows with an Nvidia GPU will display metrics in the GUI. Feel free to request that I add AMD or Apple support.

Step 3 - Download Embedding Model

The best embedding model depends on the type of text being entered into the vector database and the style of question you intend to ask. I've selected multiple models that are good, but feel free to read about each one because they're suitable for different tasks.

‼️ You must wait until the download is complete AND unpacked before trying to create the database.

Step 4 - Select Embedding Model Directory

Selects the directory of the embedding model you want to use.

Step 5 - Choose Documents for Database

Select one or more files (.pdf, .docx, .txt, .json, .enex, .eml, .msg, .csv, .xls, .xlsx, .rtf).

‼️ PDF files must already have had OCR done on them. Put in a feature request if you want to incorporate Pytesseract for OCR.

Step 6 - Create Vector Database

GPU usage will spike as the vector database is created. Wait for this to complete before querying database.

Step 7 - Start LM Studio

Open LM Studio and load a model. Click the server tab on the left side. Click "Start Server" in the server tab.

‼️ Only Llama2-based models are currently supported due to their prompt format.

Step 8 - Submit Question

Enter a question and click "submit question." The vector database will be queried and your question along with the results will be fed to LM Studio for an answer.

Step 9 - Transcribe Question Instead

Click the 'Start Record' button, speak, and then click the 'Stop' button. Paste transcription into question box and click Submit Question.

Contact

All suggestions (positive and negative) are welcome. "[email protected]" or feel free to message me on the LM Studio Discord Server.