microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.

NNI automates feature engineering, neural architecture search, hyperparameter tuning, and model compression for deep learning. Find the latest features, API, examples and tutorials in our official documentation (简体中文版点这里).

What's NEW!  

• New release: v3.0 preview is available - released on May-5-2022
• New demo available: Youtube entry | Bilibili 入口 - last updated on June-22-2022
• New research paper: SparTA: Deep-Learning Model Sparsity via Tensor-with-Sparsity-Attribute - published in OSDI 2022
• New research paper: Privacy-preserving Online AutoML for Domain-Specific Face Detection - published in CVPR 2022
• Newly upgraded documentation: Doc upgraded

Installation

See the NNI installation guide to install from pip, or build from source.

To install the current release:

$ pip install nni

To update NNI to the latest version, add --upgrade flag to the above commands.

NNI capabilities in a glance

	Hyperparameter Tuning	Neural Architecture Search	Model Compression
Algorithms	Exhaustive search Grid Search Random Heuristic search Anneal Evolution Hyperband PBT Bayesian optimization BOHB DNGO GP Metis SMAC TPE	Multi-trial Grid Search Policy Based RL Random Regularized Evolution TPE One-shot DARTS ENAS FBNet ProxylessNAS SPOS	Pruning Level L1 Norm Taylor FO Weight Movement AGP Auto Compress More... Quantization Naive QAT LSQ Observer DoReFa BNN
	Supported Frameworks	Training Services	Tutorials
Supports	PyTorch TensorFlow Scikit-learn XGBoost LightGBM MXNet Caffe2 More...	Local machine Remote SSH servers Azure Machine Learning (AML) Kubernetes Based OpenAPI Kubeflow FrameworkController AdaptDL PAI DLC Hybrid training services	HPO PyTorch TensorFlow NAS Hello NAS NAS Benchmarks Compression Pruning Pruning Speedup Quantization Quantization Speedup

Resources

• NNI Documentation Homepage
• NNI Installation Guide
• NNI Examples
• Python API Reference
• Releases (Change Log)
• Related Research and Publications
• Youtube Channel of NNI
• Bilibili Space of NNI
• Webinar of Introducing Retiarii: A deep learning exploratory-training framework on NNI
• Community Discussions

Contribution guidelines

If you want to contribute to NNI, be sure to review the contribution guidelines, which includes instructions of submitting feedbacks, best coding practices, and code of conduct.

We use GitHub issues to track tracking requests and bugs. Please use NNI Discussion for general questions and new ideas. For questions of specific use cases, please go to Stack Overflow.

Participating discussions via the following IM groups is also welcomed.

Gitter		WeChat
	OR

Over the past few years, NNI has received thousands of feedbacks on GitHub issues, and pull requests from hundreds of contributors. We appreciate all contributions from community to make NNI thrive.

Test status

Essentials

Type	Status
Fast test
Full test - HPO
Full test - NAS
Full test - compression

Training services

Type	Status
Local - linux
Local - windows
Remote - linux to linux
Remote - windows to windows
OpenPAI
Frameworkcontroller
Kubeflow
Hybrid
AzureML

Related Projects

Targeting at openness and advancing state-of-art technology, Microsoft Research (MSR) had also released few other open source projects.

• OpenPAI : an open source platform that provides complete AI model training and resource management capabilities, it is easy to extend and supports on-premise, cloud and hybrid environments in various scale.
• FrameworkController : an open source general-purpose Kubernetes Pod Controller that orchestrate all kinds of applications on Kubernetes by a single controller.
• MMdnn : A comprehensive, cross-framework solution to convert, visualize and diagnose deep neural network models. The "MM" in MMdnn stands for model management and "dnn" is an acronym for deep neural network.
• SPTAG : Space Partition Tree And Graph (SPTAG) is an open source library for large scale vector approximate nearest neighbor search scenario.
• nn-Meter : An accurate inference latency predictor for DNN models on diverse edge devices.

We encourage researchers and students leverage these projects to accelerate the AI development and research.

License

The entire codebase is under MIT license.

nni's Issues

[Trial issue] In trial code, we need to pass an environment variable to enable trial developer get current trial ID

support search space recommendation

If user provides a dataset and wants to find the best model from some candidate models, it would be better to provide proper search space for each model. To better support this scenario, the new component which recommends good search space can be added into nni.

WebUI should show more succeed trials by default

Right now, NNI WebUI only show 5 succeed trials by default. It's not enough for normal scale experiment, so suggest to change the default number from 5 to 50.

support kubeflow through a new training service

kubeflow is a popular tool for running deep learning jobs. It would be great to add a new training service which could submit kubeflow jobs on kubernetes.

no error information in some cases

If the validation in ./src/nni_manager/rest_server/restValidationSchemas.ts fails, the experiment will fail then. In this case, there is no error information, and nnictl could not find the experiment's information, such as experiment id.

~/.local/bin not added to path

When running in a docker with root user, the path ~/.local/bin is not added to $PATH
This may cause file missing issue such as
FileNotFoundError: [Errno 2] No such file or directory: 'serve'

Abnormal behavior if start 2 experiments at same time

Opened 2 terminates on a same machine and run the command
nnictl create --config ~/nni/examples/trials/mnist-annotation/config.yml
at same time

expect one of them is succeeded, another is failed, but actually encountered 2 cases

• case 1
  Both of the 2 commands show
  

• case 2
  Both of the 2 commands seem started successfully, the actually no running experiment (the webui can't be opened)

Refactor nnictl package install

refactor of nni installation

Experiment didn't stop after encountering a critical error (user environment missing tensorflow)

Create an clean Azure VM (Ubuntu 16.04), and follow the Getting Started instruction to try nni, followed the following commands:

pip3 install -v --user git+https://github.com/Microsoft/[email protected]
source ~/.bashrc
nnictl create --config ~/nni/examples/trials/mnist-annotation/config.yml

and found the below error from ~/nni/experiments/I4eOAq5y/trials/lis4f/stderr

Traceback (most recent call last):
  File "mnist.py", line 6, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

I found that it is because I missed the requirements tensorflow installed from instruction, so I tried to install tensorflow with pip3 install tensorflow, after that, when I try to run the
nnictl create --config ~/nni/examples/trials/mnist-annotation/config.yml
command again, but it promotes

Info: Checking experiment...
Error: There is an experiment running, please stop it first...
Info: You can use 'nnictl stop' command to stop an experiment!

I need manually run nnictl stop to stop the experiment, and create it again.
Why didn't the experiment stop after a critical error?

the WebUI displayed with no content

the starting process seems OK, but there is no content on the web.

ubuntu@ubuntu-Super-Server:~$ nnictl create --config nni/examples/trials/mnist-annotation/config.yml
Info: Checking experiment...
Info: Starting restful server...
Info: Checking restful server...
Info: Restful server start success!
Info: Setting local config...
Info: Success!
Info: Starting experiment...
Info: Checking web ui...
Info: Starting web ui...
Info: Starting web ui success!
Info: Web UI url: http://127.0.0.1:8080 http://192.168.123.101:8080 http://172.17.0.1:8080
Info: Start experiment success! The experiment id is ImMmO066, and the restful server post is 51188.
You can use these commands to get more information about this experiment:
commands description

1. nnictl experiment show show the information of experiments
2. nnictl trial ls list all of trial jobs
3. nnictl stop stop a experiment
4. nnictl trial kill kill a trial job by id
5. nnictl --help get help information about nnictl
6. nnictl webui url get the url of web ui

stderr.txt

nnictl tool remove start web ui related logic

Because the webui will be handled by nni manager, no need to start an external process to handle web ui, just remove all related logic from nnictl

support the feature: multiple instances of SMAC3

SMAC3 supports multiple instances (e.g., different datasets), it evaluates every configuration based on multiple runs on different instances. This is an appealing feature, as tuner could find good configurations that have good generalization.

Makefile install dependencies failed

After run make install-dependencies, the serve tool didn't install successfully.

Run an experiment at local met problem！

According to GetStarted.md ,as follws,

Install NNI through source code

git clone -b v0.1 https://github.com/Microsoft/nni.git
cd nni
chmod +x install.sh
source install.sh

I had sucessfully installed NNI ,But when I run an experiment at local ,do such commad: nnictl create --config ~/nni/examples/trials/mnist-annotation/config.yml ,It shows an error,as follws,

Info: Checking experiment...
Info: Starting restful server...
Info: Checking restful server...
Error: Restful server start failed!

Could you help me?

Rest service api/v1/nni/metric-data can not retrieve metric data for multiple trial jobs

It should return metric data for all trial jobs if job id is not specified.

[REQUEST] Maybe a list of supported tuner could be add to README.md?

Congratulations for version 0.2 !
Now the supported tuner is increasing, it might help to add a list of supported tuner for beginner.

refactor experiment stopping conditions and actions

1. nnimanager: the experiment only stops when restserver issues stop command. The experiment will not stop when reaching maxDuration/maxTrialNum, just it will not submit new trials, but can receive commands from webui and nnictl (e.g., change maxDuration/maxTrialNum, submit customized trial, etc.).
2. webui: provide button to extend experiment duration, increase maxTrialNum, stop experiment, show experiment status.

Support download log file from WebUI

nnictl tool 'create' command support user specify port

nnictl tool create command need to support user specific port, and pass the port to nni manager also by --port argument

give example code of how to run scikit-learn on nni

For example, to support regression models, use nni to choose a model as well as the hyperparameters for a given dataset.

“Hyper Parameter” is not show correctly in WebUI.

Show the total trial numbers that nni has tried in WebUI

Need Contributing.md for future contributors

Hey, I was going through #114 and trying to solve it and writing code for it. But later realized that the repository has no CONTRIBUTING document, should I go ahead and make a starter one. You guys could suggest any changes that you want in it. This would really help future contributors for reference when they need it.

nnictl crashed if config path is not correct

Use command
nnictl create --config xxx
If the path is not correct, the nnictl tool crashed
Need a friendly tip

Help for nnictl is confused

After create an experiment, I want to check the status of the experiment
Firstly, I run the command nnictl --help to see how can I check the status of the experiment,

It shows maybe I can use nnictl experiment, but after I run the command nnictl experiment, but it shows

It just a loop, I don't know how to check the status of the experiment finally.

Trial jobs need to get a unique ID value

For example, it is a common practice to do k fold data split to improve model performance. In order to do k fold split, each of my trial jobs need to train on a unique split, to do this, my trial jobs need to get a unique ID value.
In short, trial jobs need to get an Id if they need to corporate on something.
There are two possible solutions to implement this:

1. Use search space, need to add new type, such as sequence, can be implemented in either builtin or customized Tuner.
2. Add an API, such as nni.get_trial_id()

support package install through nnictl

nni will support more and more automl algorithms. It would be cumbersome to install all the supported algorithms in nni installation. We suggest to add a series of commands in nnictl to manage automl algorithms/packages. For example, nnictl package [install, uninstall, update, ...]

support enas on nni

enas (https://arxiv.org/abs/1802.03268) is Efficient Neural Architecture Search, a fast and inexpensive approach for automatic model design. It would be great to support enas on nni, which makes tuning neural architecture much easier. By running enas on nni, this kind of model design could fully leverage computation resource from training services.

support multi-phase trial tuning

A trial may need multiple phases of hyperparameter/neuralarchitecture tuning. More specifically, in phase one, the trial get some hyperparameters, then run and generate results. Then how to choose the hyperparameters in phase two should depend on the results generated in phase one. Likewise, the hyperparameters in phase three depends on the results in phase two, etc. After all the phases are completed, the trial is finished.

Not support uninstall from pip

Followed the below commands to install nni

pip3 install -v --user git+https://github.com/Microsoft/[email protected]
source ~/.bashrc

then try to uninstall it by

pip3 uninstall NNI

after uninstalled, try to run nnictl, got the below message

but expect

refactor nni.report_final_result()

In the current version, if users want to use builtin tuners, they are only allowed to report one float number through nni.report_final_result(). But users may want to report multiple numbers besides the float number for builtin tuner, and show those number on webui.
To support this requirement, for builtin tuner we also allow users to send a python object (in the current version, nni.report_final_result also allow users to report an object/structure, but builtin tuners cannot parse the object), the object should have a key named "default" and the value is the float number for tuner. Users are free to add other key/values to this object. On webui, users can choose the additional key/values to be shown in the trial status table.

GA example tuner has no config yml. Add readme.md in customized tuner folder

As the title, a known issue in release 0.1.0.

stderr path is not consistent with "trial ls" output information

support running multiple experiments on the same machine

In the current implementation, the port of the rest server in an experiment is fixed, i.e., 51188. Therefore, different experiments cannot run simultaneously on the same machine due to port conflict.

Can I restart WebUI by experiment name when I running other experiment?

support experiment status (SUCCESS, FAILED, etc.)

In the current version, nnictl supports viewing experiment profile, all the trials and their status. However, it is not convenient to know whether the experiment is finished or not both through nnictl and through webui. It would be better to add one more command in nnictl, for example, nnictl experiment status.

Remove node DeprecationWarning in log

(node:5681) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

Wrong indication for `nnictl --help`

As below shows, the usage is error.

refactor nnictl: manage builtin/installed tuner/assessor

In nni installation, we plan to only install two or three tuners and their dependencies. Other tuners/assessors will be installed when used. This feature will be supported in nnictl.

peek stdout / stderr of a running trial in the web portal

support the training service: Microsoft PAI

Microsoft PAI (https://github.com/Microsoft/pai) is a resource scheduling and cluster management platform for AI. Some Universities and companies have deploy PAI on their GPU cluster. It would be great to support a training service for PAI, allowing submitting trial jobs on PAI cluster.

nnictl tool ‘stop' command support user specify port/pid

nnictl tool stop command need to support user specific port/pid, and just stop the selected experiment

Experiment didn't stop after reaching max trial number

Try to create an experiment following the basic example

nnictl create --config ~/nni/examples/trials/mnist-annotation/config.yml

the max trial number is 1 in the config.yml

but after the only 1 trial is succeeded, the experiment didn't stop
I try to create it again, but got the below error

Got "Segmentation fault" when run make (reproducibility)

Followed the "How to contribute" document, and run the below commands:

1. git clone https://github.com/Microsoft/nni.git
2. make install-dependencies
3. make build
4. make dev-install
5. nnictl create --config ~/nni/examples/trials/mnist/config.yml

then, change some code, and run the 3rd and 4th command again, and got the below error

support SMAC3

SMAC3 (https://github.com/automl/SMAC3) is an hyperparameter tuning algorithm, which is specifically good at categorical hyperparameters. auto-sklearn also uses SMAC3 as their hyperparameter tuner. nni will support SMAC3 as a new tuner, so that users could choose SMAC3 to tune their models.

show more readable content in the generated search space file

In v0.1 and v0.2, if users use nni annotation in their trial code and use for example @nni.function_choice in their code, nni will use different numbers to designate different candidate functions from @nni.function_choice in the generated search space file. Thus, in webui, users only get the numbers instead of the real function names, which is very unfriendly. It would be great to use function names in the generated search space file.

No debugging mode

I want to contribute code to nni_manager, but don't know how to run it in debug mode.
The only way is install nni first, then replace some code in distributed folder.
Need to be more friendly for contributors.

NNI Install problem

'pip3 install -v --user git+https://github.com/Microsoft/[email protected]' met some problem. Is there any other way to install NNI ?

Installation for nni from source code on Azure B1s VM is stuck

Create a B1s type Azure VM (Ubuntu 16.04), and try to follow the command

git clone https://github.com/Microsoft/[email protected]
cd nni
make easy-install

to install nni, but it stuck, the last message is

$ react-app-rewired build --scripts-version react-scripts-ts
    Creating an optimized production build...
    Starting type checking and linting service...
    Using 1 worker with 2048MB memory limit
    ts-loader: Using [email protected] and /tmp/pip-vcid8cpq-build/src/webui/tsconfig.json
    No valid rules have been specified