🚀 Supercharge your LM Studio with a Vector Database!
‼️ Make sure to have at least 🐍Python 3.10 (haven't tested higher).
‼️ You must have both Git and git-lfs installed.
‼️ Processing.rtf
files requires that you install Pandoc first.
‼️ For any commands that begin withpython
orpip
, if you have both Python 2 and Python 3 installed, you should usePython3
orpip3
instead to make sure that the correct version of Python is used.
🪟WINDOWS INSTRUCTIONS🪟
- 🟢 Nvidia GPU ➜ install CUDA 11.8
Note that this installation is system-wide and it's not necessary to install within a virtual environment.
- 🔴 AMD GPU - Unfortunately, PyTorch does not currently support AMD GPUs on Windows (only Linux). However, it may be possible by using WSL within Windows pursuant to the instructions HERE and then access GPU-acceleration via HERE. This might also be helpful However, I do not have an AMD GPU so please let me know if you get it working with this method. If this does work for you, proceed to the instructions below on how to install my program within Linux.
- Download the latest "release" and unzip anywhere on your computer.
- Open the folder containing my repository files. Open a command prompt. Create a virtual environment:
python -m venv .
- Activate the virtual environment:
.\Scripts\activate
python -m pip install --upgrade pip
- 🟢 Nvidia GPUs:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
🔴 AMD GPU - Unfortunately, PyTorch does not currently support AMD GPUs on Windows (only Linux). However, it may be possible by using WSL within Windows pursuant to the instructions HERE and then access GPU-acceleration via HERE. However, I do not have an AMD GPU so please let me know if you get it working with this method. If this does work for you, proceed to the instructions below on how to install my program within Linux.
- 🔵 CPU only:
pip install torch torchvision torchaudio
pip install -r requirements.txt
python check_gpu.py
🐧LINUX INSTRUCTIONS🐧
- 🟢 Nvidia GPUs ➜ install CUDA 11.8
Note that this installation is system-wide and it's not necessary to install within a virtual environment.
- 🔴 AMD GPUs ➜ install ROCm version 5.6 according to the instructions.
Additionally, this repo might help, but I can't verify since I don't have an AMD GPU nor Linux.
- Download the latest "release" and unzip anywhere on your computer.
- Open the folder containing my repository files. Open a command prompt. Create a virtual environment:
python -m venv .
- Activate the virtual environment:
source bin/activate
python -m pip install --upgrade pip
- 🟢 Nvidia GPU:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
- 🔴 AMD GPU:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.6
- 🔵 CPU only:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
‼️ On Linux systems run this first before installing the requirements.txt:sudo apt-get install portaudio19-dev
andsudo apt-get install python3-dev
pip install -r requirements.txt
python check_gpu.py
🍎APPLE INSTRUCTIONS🍎
- All Macs with MacOS 12.3+ come with 🔘 Metal/MPS support, which is the equivalent of CUDA and ROCm. However, you still need to install Xcode Command Line Tools.
- Download the ZIP file containing the latest release for my repository. Inside the ZIP file is a folder holding my repository. Unzip and place this folder anywhere you want on your computer.
- Open the folder containing my repository files. Open a command prompt. Create a virtual environment:
python -m venv .
- Activate the virtual environment:
source bin/activate
python -m pip install --upgrade pip
pip install torch torchvision torchaudio
‼️ On MacOS systems run this first:brew install portaudio
pip install -r requirements.txt
python check_gpu.py
As of release 2.1+, my program allows you to speak a question and have it transcribed to the system clipboard, which you can then paste into the LM Studio question box. It uses the "faster-whisper" library, which relies upon the powerful Ctranslate2 library and the state-of-the-art "Whisper" models.
🔥TRANSCRIPTION INSTRUCTIONS🔥
Acceleration Support | Requirements | |
---|---|---|
Intel CPU | ✅ | |
AMD CPU | ✅ | |
Nvidia GPU | ✅ | CUDA |
AMD GPU | ❌ | Will default to CPU |
Apple CPU | ✅ | |
Apple Metal/MPS | ❌ | Will default to CPU |
The type of acceleration you'll be using also determines the supported quantizations (discussed below).
On Windows, simpy run ctranslate2_compatibility.exe
.
On Linux or MacOS, follow the instructions HERE.
All transcription settings can be changed in-program as of Version 2.5!
🔥USAGE INSTRUCTIONS🔥
Open a command prompt within my repository folder and activate the virtual environment:
NOTE: For 🍎Macs and 🐧Linux the command is:
source bin/activate
.\Scripts\activate
python gui.py
‼️ Only systems running Windows with an Nvidia GPU will display metrics in the GUI. Feel free to request that I add AMD or Apple support.
The best embedding model depends on the type of text being entered into the vector database and the style of question you intend to ask. I've selected multiple models that are good, but feel free to read about each one because they're suitable for different tasks.
‼️ You must wait until the download is complete AND unpacked before trying to create the database.
Selects the directory of the embedding model you want to use.
Select one or more files (.pdf
, .docx
, .txt
, .json
, .enex
, .eml
, .msg
, .csv
, .xls
, .xlsx
, .rtf
).
‼️ PDF files must already have had OCR done on them. Put in a feature request if you want to incorporate Pytesseract for OCR.
GPU usage will spike as the vector database is created. Wait for this to complete before querying database.
Open LM Studio and load a model. Click the server tab on the left side. Click "Start Server" in the server tab.
‼️ Only Llama2-based models are currently supported due to their prompt format.
Enter a question and click "submit question." The vector database will be queried and your question along with the results will be fed to LM Studio for an answer.
Click the 'Start Record' button, speak, and then click the 'Stop' button. Paste transcription into question box and click Submit Question.
All suggestions (positive and negative) are welcome. "[email protected]" or feel free to message me on the LM Studio Discord Server.