Comments (9)
@Mut1nyJD @FilipAndersson245 Hi, I just tried a by-pass to deal with the mixed precision issue, that is to cast weight and gradient to float32, update, then cast to float16. In this way the float32 burden is only applied to the weight update, but not the backward, so the computation overload would not be too much. See the link to code juntang-zhuang/Adabelief-Optimizer#31 (comment). Please let me know if you have other suggestions.
from lightweight-gan.
I guess it might be a problem with low-precision with epsilon. Because default eps=1e-8 for Adam (1e-16 for AdaBelief), this number will be just rounded to 0 in 16 bit precision. In this case, the update is pretty close to divide by 0 (or a very very small number), and the eps completely loses its impact as in float32 format.
from lightweight-gan.
Hmm, I don't really know, you should ask @FilipAndersson245
from lightweight-gan.
I have not tested Adabelief together with mixed-precision mode, maybe submit an issue at the authors github?
from lightweight-gan.
I have not tested Adabelief together with mixed-precision mode, maybe submit an issue at the authors github?
Nope not yet wanted to ask here first if anyone had some success, but will do, thank you!
from lightweight-gan.
Hi, thanks for testing with AdaBelief. AdaBelief might be incompatible with mixed-precision, because low precision might cause difference in gradient to be 0, hence division by 0. I'm not quite familiar with low-precision training, could you point to the code where low-precision takes effect? We will look into it and perhaps find some solution for the next release of AdaBelief. Thanks a lot.
from lightweight-gan.
Hmm okay I am not so sure if that it is a pure AdaBelief problem it probably is just far worse there.
But if with normal Adam it seems to become unstable as well it just takes a lot longer after a bit more than 50000 iterations it crashed out with the same error. I am still running PyTorch1.6 maybe worth upgrading to 1.7
from lightweight-gan.
But if with normal Adam it seems to become unstable as well it just takes a lot longer after a bit more than 50000 iterations it crashed out with the same error. I am still running PyTorch1.6 maybe worth upgrading to 1.7
Strangely enough this does not seem to happen when I increase the number of attention layers, maybe just pure coincidence?
from lightweight-gan.
@juntang-zhuang
Great! Sounds like a reasonable workaround to me so should work. Happy to give it a try again when I have some time.
from lightweight-gan.
Related Issues (20)
- Error(s) in loading state_dict for LightweightGAN HOT 2
- Batchnorm layer missing in the discriminator?
- showing results while training ? HOT 2
- G is showing up a negative number. What does that mean?
- Multi GPU utilization problem.
- TypeError: __init__() got an unexpected keyword argument 'hparams' HOT 1
- unable to load save model. please try downgrading the package to the version specified by the saved model HOT 7
- --load_strict=False failing to load model HOT 1
- Can't find "__main__" module (sorry if noob question) HOT 4
- loss implementation differs from paper HOT 1
- Doesn't detect CUDA from Nvidia GPU HOT 1
- Torchvision Assertion error while importing custom data
- Aim installation error HOT 4
- Discriminator Loss converges to 0 while Generator loss pretty high HOT 3
- Dont detects cuda HOT 1
- Using Lightweight GAN as a library
- Projecting generated Images to Latent Space HOT 1
- Executing with a trailing \ in the arguments sets the variable new to the truthy value '\\' and deletes all progress
- CUDA out of memory error while generating interpolations
- Minimum number data training
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lightweight-gan.