Comments (7)
Could you please provide some more details, such as your specific settings, device information, and so on?
from svd_xtend.
Thanks a lot!
I tried different resolution of bdd images but all step_loss is nan. I just use one video clip of the bdd and split the videos into some images to be fed into the model. I have tried the GTX3090 and A100.
When I use the fp32 model , the step loss is not nan but fp16'model' s loss is still nan. However, in the last block of upsample_block, query @ key.transpose(-1, -2) is too large to show nan.
My model id is "stabilityai/stable-video-diffusion-img2vid-xt", but when i tried other model is , it also doesn't work.
from svd_xtend.
My torch version is 1.13.1+cu116, and my diffusers version is 0.25.0. Even if I input the all zeros as input, the loss is also nan.
from svd_xtend.
OK, i have found the issue, the torch version should be 2.0.1 rather than 1.13.1. When I change the version of pytorch, the problem has been solved.
from svd_xtend.
Ah, I see, but in fact, I might not be able to answer why modifications to the PyTorch version would cause this issue.😢
from svd_xtend.
I upgraded the PyTorch to 2.1.2 but still has this problem, I can only train on the bf16. Any solutions?
from svd_xtend.
I upgraded the PyTorch to 2.1.2 but still has this problem, I can only train on the bf16. Any solutions?
hi did you get any solutions? I get similar problem but the loss is nan
from svd_xtend.
Related Issues (20)
- [CUDA out of memory] training in 1024 × 576 resolution in the A100 80G HOT 4
- Question:How to use the trained modules to generate a video?
- Suggestions on loading video data HOT 1
- 多卡训练报错 HOT 2
- Thank you, fine_tuned weights
- losses barely dropped with BDDx datasets??
- Error when I start the training HOT 2
- Add a short inference script loading from checkpoint? HOT 3
- How can I use the trained module
- How to support batch_size > 1? HOT 1
- 请问下BF16是怎么执行的,我执行有报错,帮看下 HOT 1
- pretrained_model_name_or_path报错 HOT 2
- RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 6 but got size 5 for tensor number 1 in the list. HOT 5
- Question about classifier guidance for image in training code HOT 3
- Exception: Could not find the transformer layer class to wrap in the model.
- after training on 512x512, the video not move always, why? HOT 2
- Training on multi GPUs with different memory size
- Provide requirements.txt by running `pip3 freeze > requirements.txt` from a working development environment HOT 1
- 36gb minimum GPU memory is required using batch size 1 and fp16 mixed precision training? HOT 9
- amazing work!will the part2's code be released?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from svd_xtend.