alonzoleeeooo / lcdg

The official code implementation of "LaCon: Late-Constraint Diffusion for Steerable Guided Image Synthesis".

Hi there 👋

🧑🏻‍🎓 I am currently a PhD student at the University of Science and Technology of China (USTC), supervised by Prof. Dong Liu.

🔍 My research direction involves a wide series of applications based on generative models such as GANs, Transformers, and diffusion models. For now, I am mainly interested in these tasks: image inpainting, text-to-image generation, and video generation.

📂 To record up-to-date resources of the aforementioned research directions, I have maintained some GitHub repos, which you can find here. Newly published papers and resources will be updated in these repos.

🤝 I am looking for long-term collaboration in ground-breaking projects in computer vision, please feel free to contact me if you are interested.

📜 You can find more information about me in the following websites.

• Google Scholar: https://scholar.google.com/citations?user=Y3NKd1wAAAAJ&hl=zh-CN&authuser=2
• Zhihu: https://www.zhihu.com/people/liu-chang-82-34-78 （@叫我Alonzo就好了）

🔥 Recent News:

• [Jun. 13th] Code and pre-trained model weights (Huggingface and ModelScope) of our paper titled "LaCon: Late-Constraint Diffusion for Steerable Guided Image Synthesis" are updated!
• [May 17th] Our paper titled "Towards Interactive Image Inpainting via Robust Sketch Refinement" is accepted by TMM 2024!

lcdg's Issues

Exception during LatentDiffusion.instantiate_first_stage

Hi @AlonzoLeeeooo,

I tried to run the training code, and encountered an error when ldm.models.autoencoder.AutoencoderKL was loading checkpoint params from pretrained first_stage_models，error detail shows below:

But when I tried debug the code, the current model showed different from the error detail above. I paste the corresponding part of the encoder model structure below:

I tried to find other works like T2I, the didn't use checkpoint for ldm.models.autoencoder.AutoencoderKL. So I wonder if you encountered this error before, and could you give some advice concerned.
Any tips or suggetions would be great appreciated. Looking forward for your reply.

Best regards.

release window of code implementation

Hey there, just wondering if there's any word on the release window for the code implementation. Any updates on the availability of the codebase would be greatly appreciated.

Loss mismatch related issues

In the paper, it is written that the loss is DM + cond, but the code only shows the cond related loss. Looking forward to hearing from you!

Extract which features in UNet? Which conference will this work be published?

Thank you for your amazing work about controlling Diffusion generation process! But I didn't find any ablation experiments about your choice of the features in UNet, which means a lot to the controlling and quality of results as for me. Are the features in Encoder too noisy to use to train the Condition Adaptor? Can you provide some more experiments? Thanks a lot!