Population Based Training [ChainerMN Feature request] about optuna

[MannyKayy]

@toshihikoyanase Thanks for the pointer to the ASHA pruner (now merged). According to the ICLR 2019 ASHA submission, ASHA > PBT in most cases so I will close out this issue.

[toshihikoyanase]

Thank you for your feature request. We also think PBT is a quite efficient method especially for optimization of neural network architectures.

I think we can implement PBT on Optuna, but it will take a substantial amount of implementation/deployment costs. This is because current Optuna does not have any mechanisms to share snapshots of models (e.g., weights of neural networks) among trials, which are required to implement 'exploit' step of PBT. The developers of PBT will need to implement the storage for model snapshots which works in parallel and distributed environments, and the users will need to deploy it by themselves.

For a meantime, we do not have plans to implement features to share model snapshots. If we receive such requests from many users, we'll prioritize the implementation of the snapshot feature and PBT.

By the way, the successive halving algorithm and hyperband, which are competitive methods with PBT, will be merged to master soon. We'll recommend them if you want to accelerate optimization.

[MannyKayy]

Yep, true. I quickly hacked together a PBT solution on top of optuna by integrating fabric and pex. It also required the addition of a class PartialTrial(BaseTrial) as some categorical hyper-parameters relate to architectural changes which must be fixed when foward sampling hyper-parameters. The storage model and database had to be modified to support distributed model version tracking. I will leave this issue open and if more people show interest in this feature, i will make some time and look into cleaning up the code and making a pull request.