ikharitonov / nnconfigs

Configurations for PyTorch training scripts

nnconfigs

nnconfigs is a small tool for simplifying PyTorch training scripts. It adds support for parametrising training configurations through simple text files, collection of training metrics and continuous saving of checkpoints.

How to install

Clone repository:

git clone https://github.com/ikharitonov/nnconfigs.git

Activate the target conda environment:

conda activate my_env

Change directory:

cd nnconfigs

Install with pip:

pip install -e .

How to use

The main way to use nnconfigs is through the command line, e.g.:

python example_training.py --model_name Model1 --dataset_name Dataset1 --configuration_name=config1 --configuration_file=/path/to/config1.txt --continue_training False

For more info, type python example_training.py --help. For the necessary format of the training script, see the provided example_training.py file.

Of course, scripts can also be run through an IDE by swapping

config = ExampleConfig(cli_args=sys.argv)

with

config = ExampleConfig(model_name="Model1", dataset_name="Dataset1", configuration_name="config1", configuration_file="/path/to/config1.txt" continue_training=False)

Lastly, you must create two additional files. First is a .py file with a custom config class, extending the nnconfigs.Config.BaseConfig. By default, it should define the directories for data, default configurations and model checkpoints, but any other custom functionality may be included (see Config.py to see what's already implemented). Second is a .txt file with the default values for all parameters used in your training script/s. In case new parameters are added, ensure that the listing in this file is complete.

TL;DR: make sure your training script looks like example_training.py and create your own custom copies of ExampleConfig.py and example_config.txt.

Extra: batch execution

This is an additional functionality implemented in two Bash scripts: batch_script.sh and start_and_monitor.sh. The main idea behind this is automating long batched training tasks (e.g. exploration of the hyperparameter space). All you need to do is prepare a folder with .txt training configurations (just as example_config.txt), a .csv file with the necessary information to execute each training script and launch the batch script, specifying where to find the .csv file, where to put the temporary logs, which anaconda environment to use and the maximum number of trainings to run concurrently at a given time.

Here is how to do this in practice.

First, create a folder with all the training configurations you want to run:

ls /path/to/configurations_folder

config1.txt
config2.txt
config3.txt
config4.txt

Create a .csv file in the following format:

cat /path/to/batch_config_info.csv

training_script_full_path,model_name,dataset_name,configuration_name,configuration_file_full_path,continue_training
/path/to/training_script1.py,MyCNNModel,MNIST_dataset,config1,/path/to/configurations_folder/config1.txt,False
/path/to/training_script1.py,ResNet-34,CIFAR10_dataset,config2,/path/to/configurations_folder/config2.txt,False
/path/to/training_script2.py,RNN,dataset2,config3,/path/to/configurations_folder/config3.txt,False
/path/to/training_script2_2.py,LSTM,dataset2,config4,/path/to/configurations_folder/config4.txt,False

Keep in mind that the first line of this file (the field names) is for your convenience, but shall not be removed because the batch script reads starting from the second line.

Before running the batch script, make sure that you are inside of nnconfigs directory and that both scripts is executable:

cd /path/to/nnconfigs
chmod +x batch_script.sh
chmod +x start_and_monitor.sh

batch_script.sh requires several arguments:

-f is the full path to the .csv file described above
-l is the full path to directory where temporary logs will be saved
-m is the maximum number of concurrently run jobs
-e is the anaconda environment to use

Then, just run the batch script:

./batch_script.sh -f /path/to/batch_config_info.csv -l ~/Desktop -m 2 -e my_env

If you want to launch it as a background process:

nohup ./batch_script.sh -f /path/to/batch_config_info.csv -l ~/Desktop -m 2 -e my_env &

Useful features

• Streamlined ingestion and training on many configurations (sets of parameters)
• Flexibility of launching through the command line
• Tidy directory structure of training outputs
• Training metrics are collected into Pandas-friendly files
• Support for continuing training from where you left off (or something crashed)
• Batch running of training scripts