countif / enas_nni

This code is for running enas on nni.

enas_nni

Description

ENAS is a algorithm for neural network search. It could learn neural network architecture via reinforcement learning. And it is much faster than NAS, designed by google.
nni is an open source AutoML toolkit for neural architecture search and hyper-parameter tuning.
This code is for running ENAS on nni.

nni: An open source AutoML toolkit for neural architecture search and hyper-parameter tuning.

link: https://github.com/Microsoft/nni

ENAS: Efficient Neural Architecture Search via Parameter Sharing

link: https://github.com/melodyguan/enas
Paper: https://arxiv.org/abs/1802.03268

How to run

1. Install dependency

Need to install nni first.
link:https://github.com/Microsoft/nni You need to choose the dev-enas branch.
run command:
python3 -m pip install --user --upgrade nni
source ~/.bashrc

2. Modify the codedir

Rnn architecture search(ptb_search):

Modify the codeDir at enas_nni/nni/examples/trials/enas/ptb_config.yml.

Cnn architecture search(cifar10_search):

macro search: Modify the codeDir at enas_nni/nni/examples/trials/enas/cifar10_macro_config.yml.
micro search: Modify the codeDir at enas_nni/nni/examples/trials/enas/cifar10_micro_config.yml.

3. Download the dataset

rnn architecture search(ptb_search): the dataset are at enas_nni/nni/examples/trials/enas/data/ptb

cnn architecture search: You need to download the cifar10 data at https://www.cs.toronto.edu/~kriz/cifar.html (python version).
And put data at enas_nni/nni/examples/trials/enas/data and rename the cifar-10-batches-py as cifar10 .
Or you could just run the script/download.sh at script folder.

4. Start run

cd ./enas_nni/nni/examples/trials/enas

ptb_search: nnictl create --config ptb_config.yml

micro_search: nnictl create --config cifar10_micro_config.yml

macro_search: nnictl create --config cifar10_macro_config.yml

Future work

We have one controller running on nni tuner and we could run multiple trials to update the controller. But we do not share the weight of each trial. Next step we will support that.

enas_nni's Issues

How to implement weight share?

It will use enas api of nni wo implement share the weight of each trial?

What is the machine configuration required to run the cifar10 example?

Every time I run the cifar10 example, the status "FAILED" appears on the nni interface. By looking at the log, I found that the current trial was Killed. How much memory and GPU memory capacity is required to run the CIFAR10 example with ENAS. Thanks!!!

My environment is configured as follows:
System: CentOS7
Graphics driver: NVIDIA GeForce GTX 1080 Ti * 2
Memory: 16G
TensorFlow: 1.13.1 GPU
Python: Python3.6

NNI APIs change since 0.9

It seems that NNI APIs change since 0.9 which I get the error below.
ModuleNotFoundError: No module named 'nni.multi_phase'
Related to: microsoft/nni#1238

How to use this on my deep neural network?

I'm looking for ways to do ENAS search on my deep neural network. Could you share some ideas on how to achieve it using your repo?

Regression task ENAS support

Is this project still under active development? When will datasets be supported for regression tasks?

Thanks

status WAITING

Is it normal that two trials status are WAITING after 40 mins?
Besides the search space is null, like ""