Question about automatic hyper-parameter tuning toolkit about elevater_toolkit_ic

[haotian-liu]

Hi @Luodian, different models have different statistics of the gradient norms depending on model architecture, pretraining approach, etc.

You can select a gradient clipping value at TRAIN.CLIP_GRAD_NORM, that is large enough so that it is only used to deal with the gradient explosion for some extremely bad batches, and for other batches, it can train normally.

To do this, we'd recommend first looking at the general statistics of your model running at first one or two epochs on several datasets, and then choose a gradient clipping value that is similar to or slightly larger than these (so that most of the parameter updates are unaffected by the gradient clipping).

Thanks.

[haotian-liu]

Hi @ZhangYuanhan-AI, thank you for your interest in our work, (and sorry for the late response).

We have encountered a similar issue in our early development stage. We found one cause is the disparity between the order of the sample sequence between different hyperparameter search runs and the final run, so that some searched hyperparameter might not be stable, and gradient explodes when it encounters some specific combination of the samples in a batch.

We fixed the sample sequence order in the latest released toolkit, and the issue has been largely alleviated, while the case can be different for each single checkpoint and its model architecture.

You can select a gradient clipping value at TRAIN.CLIP_GRAD_NORM, that is large enough so that it is only used to deal with the gradient explosion for some extremely bad batches, and for other batches, it can train normally. This has also been shown useful when we were developing our toolkit.

Thank you! And it'd be great if anyone who has a different solution for the similar issue can share and we are happy to incorporate them into our doc or toolkit :)

[ZhangYuanhan-AI]

Hi @ZhangYuanhan-AI, thank you for your interest in our work, (and sorry for the late response).

We have encountered a similar issue in our early development stage. We found one cause is the disparity between the order of the sample sequence between different hyperparameter search runs and the final run, so that some searched hyperparameter might not be stable, and gradient explodes when it encounters some specific combination of the samples in a batch.

We fixed the sample sequence order in the latest released toolkit, and the issue has been largely alleviated, while the case can be different for each single checkpoint and its model architecture.

You can select a gradient clipping value at TRAIN.CLIP_GRAD_NORM, that is large enough so that it is only used to deal with the gradient explosion for some extremely bad batches, and for other batches, it can train normally. This has also been shown useful when we were developing our toolkit.

Thank you! And it'd be great if anyone who has a different solution for the similar issue can share and we are happy to incorporate them into our doc or toolkit :)

Hi Haotian,

Thanks for your suggestion. I'll try gradient_clip first and see whether it can help.

[Luodian]

Hi we are trying to run our models on Elevator. Do you have a recommended default value for TRAIN.CLIP_GRAD_NORM to avoid explosion? like 1.0 or?

Thanks!