suryajayaraman / dl-resourced

Collection of useful DL resources found online

DL-resources

Collection of useful DL resources found online

DL courses online

1. Practical ML

Model Architectures

Transformer architectures

• Illustrated Transformers
• medium post on Tranformers
• Google AI blog
• Medium post
• Vision transformers

Resnet architectures and variants

• Simple google search

• efficientnet vs resnet

• 

• 

• 

• 

LSTM

CV strategy

• k-fold vs stratified kfold

Learning rate finder

• LR range test blogpost

LR schedulers

• https://www.jeremyjordan.me/nn-learning-rate/

• Cosine Annealing - Papers with code

• Medium post on SGDR

• With Higher learning rates, learning is faster (good until we start diverging)

• As we train longer, we tend to approach global minima, so need to reduce lr. Annealing is the part where we train with lower lr to find stable region (envisioned as plateau, where small changes in input doesnt lead to much change in loss function).

• Cyclic Learning rates - start from one end of sprectrum and increase / decrease the lr using a linear / exp / cosine like function

• warm restarts - Periodically Reseeting lr to lr_max helps us avoid overfitting regions, saddle points. Warm refers to the point that we continue to use the weights obtained after training for some time and not starting from some pre-defined initialization (random, zero etc)

• lr_max is found using the lr_range test proposed by Leslie Smith.

• Two major options

  • Cosine Annealing with warm restarts (Cosine is more aggressive annealing strategy)
  • One cycle lr (across the entire training cycle - linear / cosine annealing)
  • Generally One cycle lr is less over fitting than Cosine Annealing with warm restarts

• Lower Batch size has a regulaizing effect (less generalization error); CON - higher training time

• SGD converges little slower but Adam is fast and overfits a litte

• SWA is a method to kinda average the model as it approaches minima, SGD often settles around the local minima but takes longer to find it

• Transformers work well - Attention is all you need paper

Image Augumentation techniques

• Rand augument
• Cutmix, Mixup, gridmask kaggle post
• Cutmix and mixup on GPU/TPU by cdeotte
• GPU augumentations kaggle post

Loss functions

• Cross Entropy ???
• label smoothing CE ???
• Bitempered logistic loss function ???
• Focal loss function ???
• Taylor loss function ???

Pytorch lightning

• Tensorboard in PT-L

Optimizers

• Comparison of different Optimizers and lr_schedulers
• SGD with momentum and wieght decay, Warm restarts is generally SOTA, but takes more time to converge compared to Adam
• SGD with Stochastic Weighted averaging gives better results
• Adam converges faster but overfits lightly, different variants available -
• AdamW ???
• Ranger ???

Other topics

• snapshot learning and cyclic lr
• MoA 1st place solution
• forward selection ensemble method

Youtube channels

• Yannic kilcher
• Kaggler
• fastai