Comments (5)
Not too surprisingly "man" and "woman" classes seem to be entangled. Here is what I get for "a photo of a woman" with the three checkpoints:
from dreambooth-stable-diffusion.
Well I think indeed you should use the original model for producing reg images, as fine tuning will do some mysterious thing on the model (remember the model is also fine-tuned on the reg images). I have some thoughts: given that SD cannot generate realistic photo of human, why not use some random images of human you obtained online for regularization? Maybe you can try that as well. I feel like the original SD model will always produce some black-white faint human images with prompt like "photo of a man/woman", so maybe using external, diverse set of human photos serves as better regularization.
from dreambooth-stable-diffusion.
The first round of regularization images (from the untuned model) are pretty good/usable, so that should be fine.
I was just wondering why generating images with the regularization class noun after fine-tuning leads to such strong drift/collapse for the noun, also since you mentioned in the readme that it looks like they generate regularization images on the fly in the paper.
I'll try using more regularization images. Upon reading more closely the paper does mention that
∼200 × N “a [class noun]” samples are generated, with N being the size of the subject dataset
. So we're looking at 800-1000 recommended.
from dreambooth-stable-diffusion.
Well I think indeed you should use the original model for producing reg images, as fine tuning will do some mysterious thing on the model (remember the model is also fine-tuned on the reg images). I have some thoughts: given that SD cannot generate realistic photo of human, why not use some random images of human you obtained online for regularization? Maybe you can try that as well. I feel like the original SD model will always produce some black-white faint human images with prompt like "photo of a man/woman", so maybe using external, diverse set of human photos serves as better regularization.
have you tested that does it produce better results?
from dreambooth-stable-diffusion.
I've now tried it with more regularization images (~300, mix of curated images from the original model + internet photos, at the default 800 steps) - it seems to help a little bit with preserving diversity, but the class prior is still degraded.
Pretty impressive how well the ad hoc regularization works to generate/edit one intended new concept, but this issue limits it a bit. Not sure if they completely solved it in the Dreambooth paper either (though they're clearly aware) or just staved it off with far more regularization images.
from dreambooth-stable-diffusion.
Related Issues (20)
- %pip install -q accelerate transformers ftfy bitsandbytes==0.35.0 gradio natsort safetensors xformers
- Exception training model: 'module 'torch' has no attribute '_dynamo''.
- Interface changed for add_argparse_args() of lightning.Trainer HOT 1
- RuntimeError HOT 4
- AttributeError: module 'torch.linalg' has no attribute 'solve'
- Is there any method for loop t-step denoising to restore images and parallel speed up in stable diffusion?
- .
- This repo has many problem on windows
- cuda out of memory on RTX 24gb 3090 HOT 4
- ERROR: Failed building wheel for dlib
- Nothing Habben when Traning
- How to use DreamBooth for unconditional image synthesis.
- Questions about parameters
- ERROR: huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name':
- Implementation of metrics in the Dreambooth paper HOT 1
- RuntimeError: Error(s) in loading state_dict for LatentDiffusion: size mismatch
- Unable to train Dreambooth on Mac M1
- Dreambooth training with image captions HOT 1
- Size of the trained checkpoint (ckpt) file HOT 1
- Support for inpainting training for dreambooth?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dreambooth-stable-diffusion.