Some feedback about freeu

[ChenyangSi]

Thanks for your feedback. I tried AnimateDiff according to https://github.com/guoyww/AnimateDiff.

The following results without pick：

Original:

FreeU (different factors):

[ChenyangSi]

This is just a simple attempt according to the readme of AnimateDiff. We will be providing more results on the FreeU page and paper. We appreciate your continued interest.

[justindujardin]

Maybe it works better for cartoon-style images and animations because they naturally lack high frequencies... What were the factors that you used in the above? I can try with anime models.

@rkfg I also had poor results with FreeU, and then I started switching s1/s2/b1/b2 back to 1.0 at some point during the denoising process. The global features seem to be mostly settled early so that you can transition back to normal values between about 30% and 75% of the way through your steps.

And the results are greatly improved. I've tested it thoroughly, and most of the FreeU "fixes" are kept while still letting the fine details shine through at the end.

This is basically what I'm doing:

    steps = 30
    unet.freeu.sd21()

    def cb(step, _, __):
        if step == int(steps * 0.5):
            unet.freeu.ones()

    output = pipe(prompt, num_inference_steps=steps, callback=cb)

[dajes]

RuntimeError: cuFFT only supports dimensions whose sizes are powers of two when computing in half precision, but got a signal size of[12, 8]

To fix this you can just cast x to float for this operation in the first line of the Fourier_filter function
x_freq = fft.fftn(x.float(), dim=(-2, -1))
And cast back at the last line
return x_filtered.to(x.dtype)

[rkfg]

Maybe it works better for cartoon-style images and animations because they naturally lack high frequencies... What were the factors that you used in the above? I can try with anime models.

[rkfg]

Mario is noticeably improved, yes, but I prefer the vanilla waterfall, it's more detailed and interesting even though the contrast is a bit lower. SD 2.1 isn't that good in general, even with finetunes. Can you try on the best 1.5 models? Both cartoon and realistic? Would be interesting to see if this method can improve the output over what we can get without it.

[adhikjoshi]

Maybe it works better for cartoon-style images and animations because they naturally lack high frequencies... What were the factors that you used in the above? I can try with anime models.

@rkfg I also had poor results with FreeU, and then I started switching s1/s2/b1/b2 back to 1.0 at some point during the denoising process. The global features seem to be mostly settled early so that you can transition back to normal values between about 30% and 75% of the way through your steps.

And the results are greatly improved. I've tested it thoroughly, and most of the FreeU "fixes" are kept while still letting the fine details shine through at the end.

This is basically what I'm doing:

    steps = 30

    unet.freeu.sd21()



    def cb(step, _, __):

        if step == int(steps * 0.5):

            unet.freeu.ones()



    output = pipe(prompt, num_inference_steps=steps, callback=cb)

Can you share full code snippet?

[kadirnar]

@ChenyangSi , @rkfg
Hi, have you tested with torch.complex32? It loses all features and a gray photo appears.

[justindujardin]

Can you share full code snippet?

I describe my changes in the diffusers repo: huggingface/diffusers#5164 (comment)

[YisuiTT]

@ChenyangSi Hi, can you share how to add FreeU code in T2V, like AnimateDiff?

[ykk648]

ykk648/AnimateDiff-I2V@0842585
@YisuiTT you can refer to my codes

[YisuiTT]

@ykk648 Thank u for your codes