Comments (4)
Hi @Luodian, different models have different statistics of the gradient norms depending on model architecture, pretraining approach, etc.
You can select a gradient clipping value at TRAIN.CLIP_GRAD_NORM, that is large enough so that it is only used to deal with the gradient explosion for some extremely bad batches, and for other batches, it can train normally.
To do this, we'd recommend first looking at the general statistics of your model running at first one or two epochs on several datasets, and then choose a gradient clipping value that is similar to or slightly larger than these (so that most of the parameter updates are unaffected by the gradient clipping).
Thanks.
from elevater_toolkit_ic.
Hi @ZhangYuanhan-AI, thank you for your interest in our work, (and sorry for the late response).
We have encountered a similar issue in our early development stage. We found one cause is the disparity between the order of the sample sequence between different hyperparameter search runs and the final run, so that some searched hyperparameter might not be stable, and gradient explodes when it encounters some specific combination of the samples in a batch.
We fixed the sample sequence order in the latest released toolkit, and the issue has been largely alleviated, while the case can be different for each single checkpoint and its model architecture.
You can select a gradient clipping value at TRAIN.CLIP_GRAD_NORM
, that is large enough so that it is only used to deal with the gradient explosion for some extremely bad batches, and for other batches, it can train normally. This has also been shown useful when we were developing our toolkit.
Thank you! And it'd be great if anyone who has a different solution for the similar issue can share and we are happy to incorporate them into our doc or toolkit :)
from elevater_toolkit_ic.
Hi @ZhangYuanhan-AI, thank you for your interest in our work, (and sorry for the late response).
We have encountered a similar issue in our early development stage. We found one cause is the disparity between the order of the sample sequence between different hyperparameter search runs and the final run, so that some searched hyperparameter might not be stable, and gradient explodes when it encounters some specific combination of the samples in a batch.
We fixed the sample sequence order in the latest released toolkit, and the issue has been largely alleviated, while the case can be different for each single checkpoint and its model architecture.
You can select a gradient clipping value at
TRAIN.CLIP_GRAD_NORM
, that is large enough so that it is only used to deal with the gradient explosion for some extremely bad batches, and for other batches, it can train normally. This has also been shown useful when we were developing our toolkit.Thank you! And it'd be great if anyone who has a different solution for the similar issue can share and we are happy to incorporate them into our doc or toolkit :)
Hi Haotian,
Thanks for your suggestion. I'll try gradient_clip first and see whether it can help.
from elevater_toolkit_ic.
Hi we are trying to run our models on Elevator. Do you have a recommended default value for TRAIN.CLIP_GRAD_NORM
to avoid explosion? like 1.0
or?
Thanks!
from elevater_toolkit_ic.
Related Issues (15)
- Some details about finetuning HOT 1
- What parameters are trained in linear probe (LP) exactly?
- Problem with link to datasets HOT 7
- Missing classes in GPT-3 Knowledge source for Imagenet-1K
- Few shot learning for custom datasets
- Does the repo pick the weights that perform best in val dataset to evaluate in test dataset? HOT 2
- EvalAI taking too long HOT 5
- Recommendation with Vision Projection and Text Projection for Zero-shot learning HOT 5
- An error when running mocoV3 HOT 2
- efficiency score on parameters-efficiency track HOT 2
- AttributeError when custom model does not have a 'visual' branch HOT 1
- Why is imagenet-1k considered zero shot? HOT 1
- Log files not being created HOT 1
- compute all feature need too many memory
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elevater_toolkit_ic.