limitex / comfyui-diffusers

This repository is a custom node in ComfyUI. This is a program that allows you to use Huggingface Diffusers module with ComfyUI. Additionally, Stream Diffusion is also available.

Hi there 👋

comfyui-diffusers's Issues

How much does it require？Out of memory in workflow2.json with argument element_count 2

How much does it require？Out of memory in workflow2.json with argument element_count 2
My vram is 6GB vram, what can I do?


out of memory.json

Implement TensorRT acceleration

Implement TensorRT acceleration using this as a reference

https://github.com/cumulo-autumn/StreamDiffusion/blob/main/utils/wrapper.py

Performance

works terribly as speed compared to stable fast, doesn't offer any acceleration and locks the number of warmup steps
apprently has problems with turbo as well
no lcm, no choice or i don't know if even uses tensorrt or anything

How to use community pipeline?

Hi, I wonder if this node group allows connection to the community pipeline.
Will be nice to see a tutorial on this subject.

Issue with installation of tensorrt 9.0.1

I'm running into the following issue when running C:\ComfyUI_windows_portable\python_embeded\python.exe -m streamdiffusion.tools.install-tensorrt

`Collecting nvidia-cudnn-cu116 (from nvidia-cudnn-cu11->tensorrt_libs==9.0.1.post11.dev4->tensorrt==9.0.1.post11.dev4)
Downloading nvidia-cudnn-cu116-0.0.1.dev5.tar.gz (4.4 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "C:\Users\marst\AppData\Local\Temp\pip-install-scp8vx03\nvidia-cudnn-cu116_ccc2ef28fddc47bb88a16a018ea4061c\setup.py", line 28, in
from package_info import version
ModuleNotFoundError: No module named 'package_info'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[notice] A new release of pip is available: 23.3.2 -> 24.0
[notice] To update, run: C:\ComfyUI_windows_portable\python_embeded\python.exe -m pip install --upgrade pip
Error running command: -m pip install --pre --extra-index-url https://pypi.nvidia.com tensorrt==9.0.1.post11.dev4 --no-cache-dir
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "C:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\streamdiffusion\tools\install-tensorrt.py", line 54, in
fire.Fire(install)
File "C:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\fire\core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\fire\core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\fire\core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\streamdiffusion\tools\install-tensorrt.py", line 33, in install
run_pip(
File "C:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\streamdiffusion\pip_utils.py", line 52, in run_pip
return run_python(f"-m pip {command}", env)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\streamdiffusion\pip_utils.py", line 46, in run_python
raise RuntimeError(f"Error running command: {command}")
RuntimeError: Error running command: -m pip install --pre --extra-index-url https://pypi.nvidia.com tensorrt==9.0.1.post11.dev4 --no-cache-dir`

質問

こちら現状は複数枚を高速に生成保存するためのノードとしてとりあえず動かしてみた、という認識であっていますでしょうか。

StableDiffusionを使いたい人間はwebカメラやデスクトップの特定領域をリアルタイムに取り込んでi2iを実行する形を期待していると思うので、それが容易に使用出来るようになるノードになると嬉しいです。

img to img?

Any way to add img2img to this node?

support for Mac

getting to run on Mac requires a few tweaks...
here's an installation script I came up with, I could open a PR for you if that's helpful.

#!/bin/bash
source ../../venv/bin/activate

pip install -r requirements.txt
python python -m streamdiffusion.tools.install-tensorrt

git clone https://github.com/cumulo-autumn/StreamDiffusion.git
python -m streamdiffusion.tools.install-tensorrt

if [[ $(uname) == "Darwin" ]]; then # for MacOS / MPS
    cd StreamDiffusion
    git apply ../ignore_cuda_for_mac.patch
    pip install .
else
    python -m streamdiffusion.tools.install-tensorrt # for PC / CUDA
fi

ignore_cuda_for_mac.patch

(install issuse) Will it reinstall torch version in comfyui ? Restart ComfyUI shows Torch CUDA imcompatible, with ComfyUI python_embeded version.

install methods:

1. git clone https://github.com/Limitex/ComfyUI-Diffusers.git
2. cd ComfyUI-Diffusers
3. pip install -r requirements.txt
4. git clone https://github.com/cumulo-autumn/StreamDiffusion.git
5. python -m streamdiffusion.tools.install-tensorrt

what I did:

in this folder: ComfyUI\custom_nodes\

1. shows some issuses:
   
   

2 are the same.
3. in ComfyUI-Diffusers folder: ......\python_embeded\python.exe pip install -r requirements.txt
4. in ComfyUI-Diffusers folder: git clone https://github.com/cumulo-autumn/StreamDiffusion.git
5. in ComfyUI-Diffusers folder: ......\python_embeded\python.exe -m streamdiffusion.tools.install-tensorrt
Error:
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.1.2+cu121 with CUDA 1201 (you have 2.1.2+cpu)
Python 3.11.7 (you have 3.11.6)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details

and I restarted comfyUI,
Error:
shows Torch CUDA imcompatible

So， I use ComfyUI_windows_portable\update\update_comfyui_and_python_dependencies.bat
to reinstall torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121

restart comfyUI, ok, it works.

redo No.5 method
no Xfromers Error again.

Workflow can be imported, without Error.

So, here is my process, I hope it could help for others.

My thoughts

I think some python requirements is imcompatible, maybe xfrommers version? I'm not familiar with python, can't debug it.
https://blog.comfyui.ca/comfyui/update/2023/10/08/Weekly-update.html
the news shows comfyui remove xformers. it might be help.

Implement img2img

Implement img2img acceleration using this as a reference

https://github.com/cumulo-autumn/StreamDiffusion/blob/main/README.md

error running example

Error occurred when executing DiffusersPipelineLoader:

File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 154, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 84, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 77, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Diffusers\nodes.py", line 31, in create_pipeline
StableDiffusionPipeline.from_single_file(
File "E:\StableDiffusion\ComfyUI_windows_portable\python_embeded\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 708, in save_pretrained
save_method(os.path.join(save_directory, pipeline_component_name), **save_kwargs)
File "E:\StableDiffusion\ComfyUI_windows_portable\python_embeded\lib\site-packages\diffusers\models\modeling_utils.py", line 520, in save_pretrained
safetensors.torch.save_file(
File "E:\StableDiffusion\ComfyUI_windows_portable\python_embeded\lib\site-packages\safetensors\torch.py", line 281, in save_file
serialize_file(_flatten(tensors), filename, metadata=metadata)
File "E:\StableDiffusion\ComfyUI_windows_portable\python_embeded\lib\site-packages\safetensors\torch.py", line 475, in _flatten
return {
File "E:\StableDiffusion\ComfyUI_windows_portable\python_embeded\lib\site-packages\safetensors\torch.py", line 479, in
"data": _tobytes(v, k),
File "E:\StableDiffusion\ComfyUI_windows_portable\python_embeded\lib\site-packages\safetensors\torch.py", line 439, in _tobytes
return data.tobytes()

Works with 1.5 models, error with SDXL models or Turbo : "StableDiffusionPipeline.__init__() got an unexpected keyword argument 'text_encoder_2'"

Hi !

Thank you for making this module.

I tried the second workflow, and it runs with SD 1.5 models, but when trying to load an SDXL model or Turbo it does this error :

I installed it through ComfyUI Manager.

Edit : I guess I have my answer : cumulo-autumn/StreamDiffusion#86
It does not support SDXL for now still apparently.