Does ComfyUI support? about hidiffusion

[ShenZhang-Shin]

I will be trying to do it. If someone familiar with UI could assist, that would be even better.

[blepping]

as promised: https://github.com/blepping/comfyui_jankhidiffusion

[blepping]

i tried porting the MSW-MSA attention part to Comfy: https://gist.github.com/blepping/02e389f660112097983684a8ea8093b1

not sure i did it correctly, but the results seem reasonable and it's a noticeable speed improvement. note: barely tested at this point.

implementing the RAU-Net part seems like it would be considerably harder since i'm not sure if there's a way to replace upsample/downsample model blocks without actually patching ComfyUI itself. the code is also difficult to understand and there are conditionals that are the same in both branches. it's kind of hard to understand what it's actually doing, at least for me.

edit: this mainly seems to help with SD 1.5, i did some testing with SDXL and the performance effects seem very minor. it may not be worth using for SDXL unless you like how it affects generation quality (i feel like it may help a bit for reducing artifacts at high res for SD15 and SDXL).

[blepping]

A lot of code in RAU-Net are directly copied from diffusers impl (That is why I do not like diffuers). The main logic seem to be further increase compression / decompression by 2.

right. from what i've figured out messing with it, aside from the MSW-MSA attention part it is almost exactly Kohya Deep Shrink.

1. the cross attention part uses Torch avg_pool2d instead of bicubic.
2. the RAU downsampler part uses conv stride/dilation to downscale instead of bicubic.

for the second one, it might matter where the downscale is happening. the conv approach does seem like it produces better results than deep shrink (including with other downscale methods i've tried).

MSW-MSA attention and randomly choosing windows seems like a new idea though and from my testing works pretty well (at least with SD15).

anyway, i've finished the RAUNet part and i will probably release it tomorrow. don't get too excited though, my stuff is usually fairly janky and not too user friendly. however, that should be like 90+% of the work done so someone else could potentially take it and build on it.

it also required monkey patching ComfyUI's guts which isn't really ideal. i don't think there's any better way presently though.

[florestefano1975]

I created custom nodes to use HiDiffusion with ComfyUI.
Today I released an update that allows any SDXL checkpoint to be used.
https://github.com/florestefano1975/ComfyUI-HiDiffusion

[blepping]

@guilvalente

can you share a workflow with the right way to use this?

i updated the README and added example workflows for SD 1.5 and SDXL.

[YacratesWyh]

blepping/comfyui_jankhidiffusion#5
can anyone help this?
So far florestefano's version is just a transfer of diffuser code. We cannot use even use lora in this plugin.

[CapsAdmin]

I will be trying to do it. If someone familiar with UI could assist, that would be even better.

I tried having a quick go at this, but it's a lot of work and I think it'd be quicker if you do it. In your current diffusers code it looks like you're hijacking existing functions and injecting classes, but with comfyui you can hook onto various stages a bit more cleanly.

To get started create a directory in comfyui/custom_nodes/comfyui-hidiffusion or something similar (do not simply call it hidiffusion, otherwise if you decide to "import hidiffusion" you'll import recursively)

Then create a init.py with something like the following:

https://github.com/comfyanonymous/ComfyUI/blob/master/comfy_extras/nodes_model_downscale.py
https://github.com/comfyanonymous/ComfyUI/blob/master/comfy_extras/nodes_tomesd.py

These two nodes hook onto various stages of the process and do something similar to hidiffusion. You can literally copy paste the code from one of those nodes as a starting point into your init.py file and just rename the class and mapping names.

In case comfyui is lacking hooks to various stages or "extra_info" you'd have to modify comfyui somewhere around here:

https://github.com/comfyanonymous/ComfyUI/blob/8dc19e40d129c8ee049be7be2657458509717ba5/comfy/model_patcher.py#L125-L159
https://github.com/comfyanonymous/ComfyUI/blob/8dc19e40d129c8ee049be7be2657458509717ba5/comfy/ldm/modules/attention.py#L462-L574
https://github.com/comfyanonymous/ComfyUI/blob/8dc19e40d129c8ee049be7be2657458509717ba5/comfy/ldm/modules/diffusionmodules/openaimodel.py#L819-L890

The author would probably accept the changes if they are general enough.

[huchenlei]

i tried porting the MSW-MSA attention part to Comfy: https://gist.github.com/blepping/02e389f660112097983684a8ea8093b1

not sure i did it correctly, but the results seem reasonable and it's a noticeable speed improvement. note: barely tested at this point.

implementing the RAU-Net part seems like it would be considerably harder since i'm not sure if there's a way to replace upsample/downsample model blocks without actually patching ComfyUI itself. the code is also difficult to understand and there are conditionals that are the same in both branches. it's kind of hard to understand what it's actually doing, at least for me.

edit: this mainly seems to help with SD 1.5, i did some testing with SDXL and the performance effects seem very minor. it may not be worth using for SDXL unless you like how it affects generation quality (i feel like it may help a bit for reducing artifacts at high res for SD15 and SDXL).

A lot of code in RAU-Net are directly copied from diffusers impl (That is why I do not like diffuers). The main logic seem to be further increase compression / decompression by 2.

[guilvalente]

can you share a workflow with the right way to use this?

[blepping]

@YacratesWyh i pushed a changed today that hopefully improves the ControlNet situation. see: https://github.com/blepping/comfyui_jankhidiffusion#use-with-controlnet

this more of a bandaid type thing than a correct fix (as far as i know) so i definitely still could use help. also appreciate your post!

[AlexD81]

so sad that comfy doesnt officially support it... If I understand right, current implementations are rather hacky and may stop work on comfy updates. So can't use it, as my all projects are long term ones( But this tool would be extremely useful in comfy. Could developers contact comfy team?...