automl / autodlcomp19 Goto Github PK

Run fine-tuning experiments on these models from #5

Investigate/optimize the general scheduling/stopping criteria of model.py

Review if the dataloading we discussed still has issues and write a message to the competition organizers

Run experiments on finetuning strategies (including exploring the hyperparameters) [also specific to test dataset]

Since black had issues, include a different autoformater and also include isort again

Research and review more sophisticated online-optimization strategies (weight regularization, optimizers, lr-schedule, ...)

Add cluster compatibility for non-submission experiments, i.e. add $SLURM_ARRAY_JOB_ID to the folder name where logs are written to avoid overwriting the results.

As discussed yesterday, it would be great if we have a zoo/portfolio of pretrained models with different hyperparameters. I think considering the time left we need something that trains relatively quick. You can use WideResNet since it is a shallow model and trains for less than 3h achieving a test error of around 3% on CIFAR-10. Another reason why I chose this model is because it is trained with a step leaning rate schedule. Of course, cosine annealing might be a better choice for training a network but if we want to fine-tune from these pretrained models and keep a decent accuracy in the very first beginning we need to keep in mind that increasing the learning rate after it has decayed gets you away from that good local minima and the accuracy drops. On the other hand, if the pretrained models had a leaning rate different from 0 at the end of their training we can directly use that value when starting to fine-tune. Lets keep it like this for now, later we can experiment with cosine annealing with lr_min != 0 at the end of training.
At this point I would suggest to start running BOHB with min_budget=max_budget=200 epochs and 100 iterations. Optimize lr, weight decay and mixup interpolation factor (I will send you the code, so you can find them in the arguments. Then you just integrate this search space with the BOHB scripts you have already been running for DARTS.). The ranges can be [0.01, 1] (log scale), [1e-5, 1e-3] (log scale), [0, 1], respectively. Please make sure to save the model parameters and optimizer state after training has finished for each sampled configuration. In the end we will have 100 pretrained models on CIFAR10 that we can use later.

Review and reasearch how we should handle unbalanced data

Optimize the dataloading pipeline for speed and check for issues. Based on the script that @julien Siems wrote can you please try if it is possible to get a Dataloader object without any memory issues on PyTorch from the tf_records iterator during training (https://github.com/automl/AutoDLComp19/blob/master/src/image/before_code_structure/model.py).

Train shake-shake models on PyTorch 1.0.1, save them and add the scripts for loading the checkpoints to the codebase.

Add tensorboard to the model to visualize image model's input

Run experiments on which channel to exclude for the 4-channel dataset.
Arber: If this dataset is CMYK, we can map the (None, None, 4) tensor to RGB (None, None, 3) as pointed out here https://stackoverflow.com/questions/14088375/how-can-i-convert-rgb-to-cmyk-and-vice-versa-in-python

@anupam Kakkar, @debayan Sen @manav Madan: If the data of this dataset is not normalized and the pixel range is (0, 100) it is indeed CMYK.

Add the really small datasets from the competition starting kit to the repository.

Make an interactive plot to quickly looup the minutes or score percentage given score percentage or minutes + offset.
The input should be:
input type - percentage/minutes
t0 - The not counted seconds in the beginning
tmax - Total amount of time available
t1 - Lower bound of the region to convert
t2 - Upper bound of the region to convert

This should return the timer/percentage

• Find a way to include this as interactive graph into the Readme.md
• Implement it

Check online_meta.py if there are any logical errors

Add the option to use mixed precision training to the pipeline.

• Add the nvidia apex lib to the environment.yaml for automatic install
• Add the necessary files to the config.hjson the library needs to function
• Add an option to the config.hjson to use/not use mixed precision training

• Check capabilities
• Tick off 1

Identify which datasets transfer well to each other.

Implement random initialization (and not loading from pretrained). We need this to determine online at which point we want to evaluate our model.

ResNet-101, ResNet-152, DenseNet-161, ResNeXt-50-32x4d, ResNeXt-101-32x8d from torchvision, to the codebase so they can be easily called and fine-tuned.

Create a preprocessing and augmentation pipeline