no image2Image support ? about hidiffusion

[fkjkey]

but I used XL and got it
NameError: name 'scale_lora_layers' is not defined @ShenZhang-Shin

[radames]

thanks @ShenZhang-Shin , amazing, put it together as a Space demo here https://huggingface.co/spaces/radames/Enhance-This-HiDiffusion-SDXL using the latest controlnet SDXL https://huggingface.co/TheMistoAI/MistoLine

QuickTime.Player.mp4

[ShenZhang-Shin]

thanks @ShenZhang-Shin , amazing, put it together as a Space demo here https://huggingface.co/spaces/radames/Enhance-This-HiDiffusion-SDXL using the latest controlnet SDXL https://huggingface.co/TheMistoAI/MistoLine

QuickTime.Player.mp4

Nice work. However, I find that the output image is not as good as my code output, such as Lara. There might be some problems to fix.

[ShenZhang-Shin]

very cool! I'm using this new controlnet net model and a custom sobel operator to generate the canny , https://huggingface.co/TheMistoAI/MistoLine , it would produce different outputs for sure

Wow, A new controlnet that Controls Every Line! Thanks for sharing. I will be going to use it with hidiffusion.

[ShenZhang-Shin]

Please wait, we will soon support it.

[Bluebang]

+1

[radames]

this will be amazing! just testing with pure controlnet without the reference image is suboptimal

[fkjkey]

Is controlnet only supported in the XL version?
I used version 1.5 and got it
File "/root/anaconda3/envs/stable_fast/lib/python3.10/site-packages/hidiffusion/hidiffusion.py", line 383, in call
self.check_inputs(
TypeError: StableDiffusionControlNetPipeline.check_inputs() takes from 4 to 13 positional arguments but 17 were given

[ShenZhang-Shin]

Yes, We now only support XL+controlnet.

[ShenZhang-Shin]

but I used XL and got it NameError: name 'scale_lora_layers' is not defined @ShenZhang-Shin

Do you run hidiffusion with diffusers==0.27.0 or diffusers==0.25.0 ? We have tested the two versions and it works fine.
scale_lora_layers belongs to diffusers. Other versions may meet incompatibility

[ShenZhang-Shin]

Now we support image2image, welcome to try it to generate impressive images !
Here is an example.
Given a blurry image, hidiffusion can output a 2K clear image.

[fkjkey]

但是我使用了XL并得到了NameError：未定义名称“scale_lora_layers”

您是否使用 diffusers==0.27.0 或 diffusers==0.25.0 运行 hidiffusion ？我们已经测试了这两个版本，它运行良好。 scale_lora_layers属于扩散器。其他版本可能不兼容

I use XL-lightning, I wonder if that's the reason

[ShenZhang-Shin]

Please provide your code with XL-lightning, I will reproduce it and check the reason

[ShenZhang-Shin]

thanks @ShenZhang-Shin , amazing, put it together as a Space demo here https://huggingface.co/spaces/radames/Enhance-This-HiDiffusion-SDXL using the latest controlnet SDXL https://huggingface.co/TheMistoAI/MistoLine

QuickTime.Player.mp4

The lara output with the same prompt and guidance scale on my machine. The output is better, especially in the background.

prompt: photography of lara croft 8k high definition award winning
negative prompt: underexposed, poorly drawn hands, duplicate hands, bad limbs, overexposed, bad art, beginner, amateur, abstract, disfigured, deformed
strength: 0.99
controlnet_conditioning_scale: 0.5
guidance_scale: 8.5

[radames]

very cool! I'm using this new controlnet net model and a custom sobel operator to generate the canny , https://huggingface.co/TheMistoAI/MistoLine , it would produce different outputs for sure

[fkjkey]

Please provide your code with XL-lightning, I will reproduce it and check the reason

class Xl_make():
  def __init__(self):
      self.controlnet_canny = ControlNetModel.from_pretrained(
          "/home/lora_test/model/canny_xl", use_safetensors=True, torch_dtype=torch.float16
      )
      self.pipe = StableDiffusionXLControlNetImg2ImgPipeline.from_pretrained(
          "/home/lora_test/model/dreamshaper_xl_lightning",
          torch_dtype=torch.float16,
          controlnet=self.controlnet_canny,
          use_safetensors=True,
      ).to('cuda')
      self.pipe.scheduler = DPMSolverMultistepScheduler.from_config(
        self.pipe.scheduler.config,
        algorithm_type="sde-dpmsolver++"
      )
      apply_hidiffusion(self.pipe)
  
  def make(self, image):
      width, height = image.size
      if width < 832 or height < 832:
         image = image.resize((width*2, height*2))
         width, height = image.size
      if width > 832 or height > 832:
         image = image.resize((int(832 / max(width, height) * width), int(832 / max(width, height) * height)))
      width, height = image.size
      image = image.resize((width, height))

      canny_img = np.array(image)
      low_threshold = 100 
      high_threshold = 200
      canny = cv2.Canny(canny_img, low_threshold, high_threshold)
      canny = Image.fromarray(canny)

      new_img = self.pipe(
         prompt='oil painting,Masterpiece,best quality,nature,in a meadow,lake,sky'
         negative_prompt='bad quality, worst quality, text, signature, watermark, extra limbs,blurred, watermark, signature, low contrast, low resolution',
         image=image,
         control_image=canny,
         guidance_scale=4,
         strength=0.6
         num_inference_steps=10
         controlnet_conditioning_scale=1.0,
         generator=torch.manual_seed(42),
         control_guidance_start=0.0,
         control_guidance_end=0.6,
         clip_skip=2,
      ).images[0]
      new_img.save('/home/sdmade_material/new_img/new1.jpg')

      new_img.save(f'/home/sdmade_material/new_img/new1.5.jpg')

[ShenZhang-Shin]

Please provide your code with XL-lightning, I will reproduce it and check the reason

class Xl_make():
  def __init__(self):
      self.controlnet_canny = ControlNetModel.from_pretrained(
          "/home/lora_test/model/canny_xl", use_safetensors=True, torch_dtype=torch.float16
      )
      self.pipe = StableDiffusionXLControlNetImg2ImgPipeline.from_pretrained(
          "/home/lora_test/model/dreamshaper_xl_lightning",
          torch_dtype=torch.float16,
          controlnet=self.controlnet_canny,
          use_safetensors=True,
      ).to('cuda')
      self.pipe.scheduler = DPMSolverMultistepScheduler.from_config(
        self.pipe.scheduler.config,
        algorithm_type="sde-dpmsolver++"
      )
      apply_hidiffusion(self.pipe)
  
  def make(self, image):
      width, height = image.size
      if width < 832 or height < 832:
         image = image.resize((width*2, height*2))
         width, height = image.size
      if width > 832 or height > 832:
         image = image.resize((int(832 / max(width, height) * width), int(832 / max(width, height) * height)))
      width, height = image.size
      image = image.resize((width, height))

      canny_img = np.array(image)
      low_threshold = 100 
      high_threshold = 200
      canny = cv2.Canny(canny_img, low_threshold, high_threshold)
      canny = Image.fromarray(canny)

      new_img = self.pipe(
         prompt='oil painting,Masterpiece,best quality,nature,in a meadow,lake,sky'
         negative_prompt='bad quality, worst quality, text, signature, watermark, extra limbs,blurred, watermark, signature, low contrast, low resolution',
         image=image,
         control_image=canny,
         guidance_scale=4,
         strength=0.6
         num_inference_steps=10
         controlnet_conditioning_scale=1.0,
         generator=torch.manual_seed(42),
         control_guidance_start=0.0,
         control_guidance_end=0.6,
         clip_skip=2,
      ).images[0]
      new_img.save('/home/sdmade_material/new_img/new1.jpg')

      new_img.save(f'/home/sdmade_material/new_img/new1.5.jpg')

I see, man. I will reproduce it and fix the problem