hangover3832 / comfyui-hangover-nodes

Various nodes for ComfyUI

Custom nodes for ComfyUI

ComfyUI is the awesome stable diffusion GUI and backend.

Please note that this repository is currently a (learning) work in progress and might change anytime. It has been tested in Windows 10 only so far.

Updates

• Save Image now has only 1 button to include/exclude metadata. You now can disable saving the image, thus it acts as an image preview only. Note that you can still save the image, by right clicking on it, that will include or exclude the workflow according to the setting. 🆕
• kosmos-2 interrogator now outputs a mask indicating the recognized elements in the image. 🆕
• "Image Scale To Bounding Box" now can pad the image to its given box size (examples). 🆕
• New node: Make inpainting model. 🆕
• Kosmos-2 update: Ability to strip out the prompt from the response (credit to za-wa-n-go). 🆕
• Kosmos-2 now supports manually downloadad model folder location. 🆕

Nodes overview:

• Kosmos-2: Grounding Multimodal Large Language Models to the World
• Stable Diffusion Privacy: Save Image with or without Metadata
• Scale an Image To A Bounding Box
• Easily make an inpainting version of any SD1.5 model

---

Node: Microsoft kosmos-2 for ComfyUI

An implementation of Microsoft kosmos-2 text & image to text transformer .

This node takes a prompt that can influence the output, for example, if you put "Very detailed, an image of", it outputs more details than just "An image of". kosmos-2 is quite impressive, it recognizes famous people and written text in the image:

kosmos-2 output: An image of Donald Trump giving the peace sign with the words "Make America Great Again" written next to him.

At the first start, the kosmos-2 model files will be downloaded from huggingface. Please be patient. The model file is about 6GB in size. There is a cpu/gpu selector, but be aware that the model will eat up about 6GB of your precious VRAM in gpu mode! Alternatively, the model can be downloadad manually. Place the files in a folder named "kosmos-2-patch14-224" under the ./ComfyUI/models/kosmos2 folder. ./ComfyUI/models/kosmos2/kosmos-2-patch14-224 must contain the following files:

added_tokens.json
config.json
generation_config.json
model.safetensors
preprocessor_config.json
sentencepiece.bpe.model
special_tokens_map.json
tokenizer.json
tokenizer_config.json

The kosmos2 base folder can also be configured in extra_model_paths.yaml

See example outputs and workflows

Also see Moondream, Recognize Anything Model

---

Node: Save Image w/o Metadata

With this custom save image node, you can preview or save, include or exclude the ComfyUI workflow metadata in the image. It is a derivation of ComfyUI's built-in save image node. Note that you can always right click on the image to save, it will also include the workflow if activated.

---

Node: Scale Image To Bounding Box

This node scales an input image into a given box size, whereby the aspect ratio keeps retained. The image can also be padded to the full box size with an arbitrary color.

See example outputs and workflows

---

Node: Make Inpainting Model for SD1.5

This node easy creates an inpainting version of any SD1.5 model on the fly. No need to have GB's of inpainting models laying on your drive. This is very useful for any kind of inpainting nodes like detailers. Make sure you have the original SD1.5 models from RunwayML in your models folder:

• v1-5-pruned-emaonly.ckpt orv1-5-pruned-emaonly.safetensors
• sd-v1-5-inpainting.ckpt

They are needed for the calculation.

See examples

---

Installation

Unzip or git clone this repository into ComfyUI/custom_nodes folder and restart ComfyUI.

comfyui-hangover-nodes's Issues

Proposal

Is there a way to avoid including prompt in the response?
I am considering using the response directly as a gen image prompt, but it is difficult to use because it contains prompt.

MsKosmos2 starts loading the model from the Internet with each new launch

This makes it very difficult to work. Fix please

Each time "ms_kosmos2" is used, an internet connection is required, which results in the inability to utilize this feature offline.

Thank you for your selfless dedication; I hope we can resolve this issue.

Off topic: What is giterhub?

Can somebody tell me what/who giterhub is, how its related to github and why i am a "member" there?

AttributeError: 'MsKosmos2' object has no attribute 'model'

ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "X:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 155, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "X:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 85, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "X:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 78, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "X:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Hangover-Nodes\ms_kosmos2.py", line 51, in interrogate
if (self.model == None) or (self.processor == None) or (self.modelname != huggingface_model) or (device != self.device):
^^^^^^^^^^
AttributeError: 'MsKosmos2' object has no attribute 'model'

'MsKosmos2' object has no attribute 'model'

I really wanted to try this out, but I couldn't get it running. I'm missing the model, and it's not downloading automatically. If I downloaded itmanually, where should I put it?

Error occurred when executing MS kosmos-2 Interrogator:

'MsKosmos2' object has no attribute 'model'

File "***ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "***ComfyUI_windows_portable\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "***ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "***ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Hangover-Nodes\ms_kosmos2.py", line 49, in interrogate
if (self.model == None) or (self.processor == None) or (self.modelname != huggingface_model) or (device != self.device):
^^^^^^^^^^

ImportError:

ImportError: cannot import name 'Kosmos2Config' from 'transformers' (A:\uAI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers_init_.py)

Cannot import A:\uAI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Hangover-Nodes module for custom nodes: cannot import name 'Kosmos2Config' from 'transformers' (A:\uAI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers_init_.py)

issue: config fail

After installation, the following error occurs upon the first launch.

Traceback (most recent call last):
  File "/mnt/teratera/git/ComfyUI/nodes.py", line 1810, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/mnt/teratera/git/ComfyUI/custom_nodes/ComfyUI-Hangover-Nodes/__init__.py", line 1, in <module>
    from .ms_kosmos2 import NODE_CLASS_MAPPINGS
  File "/mnt/teratera/git/ComfyUI/custom_nodes/ComfyUI-Hangover-Nodes/ms_kosmos2.py", line 17, in <module>
    model = AutoModelForVision2Seq.from_pretrained("microsoft/kosmos-2-patch14-224")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 434, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 873, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 579, in __getitem__
    raise KeyError(key)
KeyError: 'kosmos-2'

Cannot import /mnt/teratera/git/ComfyUI/custom_nodes/ComfyUI-Hangover-Nodes module for custom nodes: 'kosmos-2'

Bbox list output

kosmos2 can output bboxes to corresponding keywords that were recognized in the input image, could be useful in ComfyUI.

Feel free to contribute.

can't be downloaded

The model can't be downloaded automatically. You don't specify which folder it should be placed in when manually downloading it. There's only a model name, and no matter where it's placed, it will be hard to find later on. There are too many files with this name.