bbc-mc / sdweb-clip-changer

Add setting to change CLIP. Extension for AUTOMATIC1111 web ui

Hi, I'm bbc_mc (a.k.a bbc-mc on github)

sdweb-clip-changer's Issues

Does this support in the webui api use

@bbc-mc How to use sdweb-clip-changer in webui api

Request for support for stable diffusion forge.

The current version (fecdf96) does not seem to work when installed on the forge extension.
I use both a1111 and forge and go back to a1111 when needed, but would like to use the clip changer in forge as well.

Does this support A1111 v1.6.x, and some other questions.

Hi,

Just wondering if this extension supports A1111 v1.6.x?

So far I have seen no sign of the extension downloading the clip I selected (openai/clip-vit-large-patch14-336).

I had to switch from the old one-click installer/Launcher to Stability Matrix, as the old one is no longer being developed for some reason, and I question whether or not Stability Matrix properly reports all CLI output, as it uses its own console rather than just opening a command prompt.

Would it be possible for you to add a "force CLIP model download" button, so that we can be sure the extension has actually fetched our selected CLIP model?

thanks in advance for any help.

Device confusion

I ran into some trouble using this with Counterfeit V3.0, which requires openai/clip-vit-large-patch14-336.

  CLIPTextModel was on cpu
  CLIPTextModel applied: openai/clip-vit-large-patch14-336
  CLIPTokenizer applied: openai/clip-vit-large-patch14-336
Applying scaled dot product cross attention optimization.
Model loaded in 3.8s (load weights from disk: 0.1s, create model: 0.4s, apply weights to model: 0.4s, apply half(): 0.3s, load VAE: 0.2s, scripts callbacks: 2.3s).
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
CLIP Changer.on_app_started done.
Startup time: 8.7s (import torch: 0.8s, import gradio: 0.6s, import ldm: 0.4s, other imports: 0.4s, load scripts: 0.5s, load SD checkpoint: 3.9s, scripts before_ui_callback: 0.1s, create ui: 1.9s).
Error completing request
Arguments: ('task(ll4q9oh1fby7wn2)', '(masterwork, best quality, ultrarealistic:0.5). Energetic girl on the train', 'EasyNegativeV2', [], 20, 15, False, False, 1, 1, 7, 3870221604.0, -1.0, 0, 0, 0, False, 512, 768, False, 0.55, 2, 'Lanczos', 20, 0, 0, [], 0, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0) {}
Traceback (most recent call last):
  File "/home/svein/AI/sd-bot/modules/call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "/home/svein/AI/sd-bot/modules/call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "/home/svein/AI/sd-bot/modules/txt2img.py", line 56, in txt2img
    processed = process_images(p)
  File "/home/svein/AI/sd-bot/modules/processing.py", line 503, in process_images
    res = process_images_inner(p)
  File "/home/svein/AI/sd-bot/modules/processing.py", line 642, in process_images_inner
    uc = get_conds_with_caching(prompt_parser.get_learned_conditioning, negative_prompts, p.steps, cached_uc)
  File "/home/svein/AI/sd-bot/modules/processing.py", line 587, in get_conds_with_caching
    cache[1] = function(shared.sd_model, required_prompts, steps)
  File "/home/svein/AI/sd-bot/modules/prompt_parser.py", line 140, in get_learned_conditioning
    conds = model.get_learned_conditioning(texts)
  File "/home/svein/AI/sd-bot/repositories/stable-diffusion-stability-ai/ldm/models/diffusion/ddpm.py", line 669, in get_learned_conditioning
    c = self.cond_stage_model(c)
  File "/home/svein/AI/sd-bot/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/svein/AI/sd-bot/modules/sd_hijack_clip.py", line 229, in forward
    z = self.process_tokens(tokens, multipliers)
  File "/home/svein/AI/sd-bot/modules/sd_hijack_clip.py", line 254, in process_tokens
    z = self.encode_with_transformers(tokens)
  File "/home/svein/AI/sd-bot/modules/sd_hijack_clip.py", line 302, in encode_with_transformers
    outputs = self.wrapped.transformer(input_ids=tokens, output_hidden_states=-opts.CLIP_stop_at_last_layers)
  File "/home/svein/AI/sd-bot/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/svein/AI/sd-bot/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 816, in forward
    return self.text_model(
  File "/home/svein/AI/sd-bot/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/svein/AI/sd-bot/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 712, in forward
    hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids)
  File "/home/svein/AI/sd-bot/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/svein/AI/sd-bot/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 227, in forward
    inputs_embeds = self.token_embedding(input_ids)
  File "/home/svein/AI/sd-bot/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/svein/AI/sd-bot/modules/sd_hijack.py", line 234, in forward
    inputs_embeds = self.wrapped(input_ids)
  File "/home/svein/AI/sd-bot/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/svein/AI/sd-bot/venv/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/home/svein/AI/sd-bot/venv/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

...as you can see, the text model needs to be moved to the GPU.

However, like you can probably guess, I modified your code to print sd_model.cond_stage_model.transformer.device prior to swapping the text model. Which printed "cpu". Nevertheless, swapping to .to('cuda') fixed the problem. I'm really not sure how that can happen -- maybe there's some code elsewhere in A1111 to move it, and swapping the model breaks that somehow?