deep500 / deep500

A Deep Learning Meta-Framework and HPC Benchmarking Library

Deep500: A Deep Learning Meta-Framework and HPC Benchmarking Library

(or: 500 ways to train deep neural networks)

Deep500 is a library that can be used to customize and measure anything with deep neural networks, using a clean, high-performant, and simple interface. Deep500 includes four levels of abstraction: (L0) Operators (layers); (L1) Network Evaluation; (L2) Training; and (L3) Distributed Training.

Using Deep500, you automatically gain:

• Operator validation, including gradient checking for backpropagation
• Statistically-accurate performance benchmarks and plots
• High-performance integration with popular deep learning frameworks (see Supported Frameworks below)
• Running your operator/framework/optimizer/communicator/... with real workloads, alongside existing environments
• and much more...

Installation

Using pip: pip install deep500

Usage

See the tutorials.

Requirements

• Python 3.5 or later
• Protobuf (sudo apt-get install protobuf-compiler libprotoc-dev)
• For plotted metrics: matplotlib
• For distributed optimization:
  • Any MPI implementation (OpenMPI, MPICH, MVAPICH etc.)
  • mpi4py Python package

Supported Frameworks

• Tensorflow
• Pytorch
• Caffe2

Reference

If you use this meta-framework please cite it as:

@inproceedings{deep500,
  author={T. Ben-Nun and M. Besta and S. Huber and A. N. Ziogas and D. Peter and T. Hoefler},
  title={{A Modular Benchmarking Infrastructure for High-Performance and Reproducible Deep Learning}},
  year={2019},
  month={May},
  publisher={IEEE},
  note={The 33rd IEEE International Parallel \& Distributed Processing Symposium (IPDPS'19)},
}

Contributing

Deep500 is an open-source, community driven project. We are happy to accept Pull Requests with your contributions!

License

Deep500 is published under the New BSD license, see LICENSE.

deep500's Issues

run "tf_native_training.py" error

when I executed the file "tf_native_training.py", appeared error:
Traceback (most recent call last):
File "tf_native_training.py", line 29, in
executor = d5tf.TensorflowNativeGraphExecutor(cost, pred.name)
File "/THL5/home/wql17/anaconda3/lib/python3.6/site-packages/deep500/frameworks/tensorflow/tf_graph_executor.py", line 167, in init
self.network.add_output(output_node.name)
AttributeError: 'str' object has no attribute 'name'

How can I benchmark the gradients compression method in Deep500?

I need to test my gradients compression algorithm, but the tutorial does not include level3, would you give me some instruction, thank you!

ifstat path

Hi,

ifstat path is assumed to be /usr/sbin/ifstat, which is not necessarily always the case. A better option would be to use shutil.which() to determine the path (link to docs), which is part of Python 3.3+.

Alessandro