Comments (7)
Are you using Amp? If so, we're about to include some optimizations targeting Pytorch 1.0 (based on the fact that their C++ API is now stable) which will greatly reduce the cost of checking for infs and nans (necessary to implement dynamic loss scaling) performed as part of the optimization step.
I didn't write the FusedAdam optimizer, but I'll pass along your request.
from apex.
Thanks, currently I'm using FP16_Optimizer instead of Amp. My usecase is standard imagenet training, using AdamW instead of the SGD. I understand from the documentation that FP16_Optimizer is best, or should I be using Amp?
Basically I'm after highest throughput. Lower GPU memory usage is also nice, but throughput is my chief concern.
I gave dynamic loss scaling a go with FP16_Optimizer, it does seem to be quite a bit slower than using static loss scaling. I'm glad to hear it'll become faster. I'm just having a read of the mixed precision training paper again (https://arxiv.org/pdf/1710.03740.pdf), would it otherwise be possible to monitor the mean & standard deviation of the gradients? Perhaps you could choose a loss scale value such that the distribution stays within acceptable bounds. Such a check you could also probably just perform once per epoch?
from apex.
Hi @yaysummeriscoming ,
We are looking into adamW as well as some general improvement to fusedAdam.
Since the fused adam does not support weight decay at all. Do you think we should also support original pytorch adam weight decay?
from apex.
@FDecaYed , that's great news. Perhaps for completeness it might be a good idea, but in practice I'm always using the AdamW formulation for CNNs. I've found it to always be superior.
Edit: I have a python reference implementation, if that's helpful?
from apex.
@FDecaYed Any update on this? I'd also like a fusedAdamW.
from apex.
@yaysummeriscoming @EvenOldridge
The latest apex.optimizers.FusedAdam already support AdamW. You can use it with optimizers.fp16_optimizer(or amp O2 with manual cast? @mcarilli to confirm)
What it does not support is original adam with L2.
Sorry for not seeing and replying. I made the change ~ a week after last replied here and forgot to update this thread.
from apex.
Thanks for the update & putting this in!
from apex.
Related Issues (20)
- Unable to install Apex HOT 2
- Release GIL when calling C extensions
- unable to install HOT 3
- Unable to install Apex on Linux(debian) with CUDA 12.1 and torch 2.2.2 HOT 2
- Unknown CUDA arch (compute) or GPU not supported error while installing on docker ubuntu with cuda 12.1
- Install set.up
- Setting up Apex and get this error: ModuleNotFoundError: No module named 'torch' HOT 8
- Unable to install Apex
- Slow Performance with "Exhaustive Search" Permutation Strategy for Channel Pruning in CNN
- Unsuccessful installation of apex library. (Preparing metadata (pyproject.toml) did not run successfully.) HOT 1
- No CUDA runtime is found, using CUDA_HOME='/home/shengjieyi/cuda1108' .
- Installation with Cuda extentions is failling
- Unsupported NVHPC compiler found. nvc++ is the only NVHPC compiler that is supported.
- install bug with pytorch2.0.1
- How to improve training performance with Apex package
- loss scale
- No module named 'amp_C'
- Gradient Overflow with Specific GPU Combinations in Multi-GPU Setup (NVIDIA RTX 3090)
- Discrepancy with Optimizer States and Model State Dict when using store_param_remainders==True HOT 8
- ASP Automatic Sparsity forward function For Loop Error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from apex.