A simple local text generator for ComfyUI utilizing ExLlamaV2.
Clone the repository to custom_nodes
:
git clone https://github.com/Zuellni/ComfyUI-ExLlama-Nodes custom_nodes/ComfyUI-ExLlamaV2-Nodes
Install the requirements:
pip install -r custom_nodes/ComfyUI-ExLlamaV2-Nodes/requirements.txt
On Windows install one of the precompiled wheels instead:
pip install https://github.com/turboderp/exllamav2/releases/download/v0.0.xx/exllamav2-0.0.xx+cuXXX-cpXXX-cpXXX-win_amd64.whl
Check which one you need with:
python -c "import sys, torch; print(f'cu{torch.version.cuda.replace('.', '')}-cp{sys.version_info[0]}{sys.version_info[1]}')"
Caution
If you see any ExLlamaV2-related errors while the nodes are loading, try to install it manually following the official instructions.
Only EXL2 and 4-bit GPTQ models are supported. You can find a lot of them on Hugging Face. Refer to the model card in each repository for details about quant differences and instruction formats.
To use a model with the nodes, you should clone its repository with git or manually download all the files and place them in models/llm
.
For example, if you'd like to download Mistral-7B, use the following command:
git clone https://huggingface.co/LoneStriker/Mistral-7B-Instruct-v0.2-5.0bpw-h6-exl2-2 models/llm/mistral-7b-exl2-b5
Tip
You can add your own llm
path to the extra_model_paths.yaml file and place the models there instead.
Loader | Loads models from the llm directory. |
|
cache_bits | Lower value equals lower VRAM usage but also impacts generation speed. | |
max_seq_len | Max context, higher value equals higher VRAM usage. 0 will default to config. |
|
Generator | Generates text based on the given prompt. Refer to text-generation-webui for parameters. | |
unload | Unloads the model after each generation. | |
single_line | Stops the generation on newline. | |
max_tokens | Max new tokens, 0 will use available context. |
|
Previewer | Displays generated text in the UI. | |
Replacer | Replaces variable names enclosed in brackets, eg [a] , with their values. |
The example workflow is embedded in the image below and can be opened in ComfyUI.