microsoft / azure-iot-edge-cv-model-samples

An end-to-end IoT Edge solution performing object detection (YOLOv3)

Bringing Computer Vision models to the Edge with Azure IoT Edge samples [UPDATED NOV 2019]

Overview

This repository hosts the code samples for the Bringing Computer Vision models to the Edge with Azure IoT Edge - A guide for developers and data scientists.

The objective of this guide is to walk you through an end-to-end AI object detection on a Raspberry Pi 3 via a series of modules that are entirely customizable with ease. This guide is designed for developers as well as data scientists who wish to easily put their AI models in practice on edge devices without focusing too much on the deployment.

[UPDATED NOV] This new version features a fully operational MLOps implementation of the above IoT solution. Check out MLOps/ as well as the associated section in the updated guide

From the training of the YOLOv3 object detection to the deployment on the Raspberry Pi 3, you will have a wide overview of how to build an IoT device performing computer vision models.

Contents

• Azure ML Training: contains a notebook to train the state-of-the-art object detection YOLOv3 based on this Keras implementation repository with Azure Machine Learning.
• IoT: contains the IoT solution presented in the guide as well as a notebook to quickly set up an Azure IoT Hub via az commands.
• MLOps: contains an end-to-end MLOps implementation of the IoT solution above from the training of YOLOv3 on the VOC dataset to the deployment in a dev-qa-prod environment with Azure DevOps.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

azure-iot-edge-cv-model-samples's Issues

Getting std::bad_alloc error when running Object Detector module on Rasberry Pi 3 B+

Hi, I am facing this issue std::bad_alloc when running the ObjectDetector module on Raspberry Pi 3 B+ and haven't had much luck so far.

I am using the same onnxruntime-0.5.0-cp35-cp35m-linux_armv7l.whl file to install ONNX runtime on the container as I was not able to compile a new version from the latest ONNX repo.
It kept crashing when I tried to do the docker build for ARM 32 from their docker file.

I set up the same ObjectDetector module in my solution and deployed to the edge device.
ObjectDetector module crashes in predict_yolov3.py when it initializes onnxruntime.

Complete error is below:
Traceback (most recent call last): File "app.py", line 55, in <module> model.initialize() File "/app/predict_yolov3.py", line 51, in initialize session = onnxruntime.InferenceSession(self.model) File "/usr/local/lib/python3.5/dist-packages/onnxruntime/capi/session.py", line 29, in __init__ self._sess.load_model(path_or_bytes) RuntimeError: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: std::bad_alloc root@raspberrypi:/home/pi# Loading model... File "/app/predict_yolov3.py", line 51, in initialize session = onnxruntime.InferenceSession(self.model) File "/usr/local/lib/python3.5/dist-packages/onnxruntime/capi/session.py", line 29, in __init__ self._sess.load_model(path_or_bytes) RuntimeError: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: std::bad_alloc
Not sure if I am doing anything different to this solution. Maybe its something to do with the memory allocation on the edge device. But my other modules on the device work fine.

Can you please point me in the right direction or let me know if there is any update for predict_yolov3.py file?
Thanks

Issue with training model with Azure Machine Learning.

I followed the guide on training model in Azure Machine Learning and got an error after 8 hours of training. From the logs I can see that it completet the first 50 epochs and crashed when it tried to start 51/60. The error messages are listed below.

Any ideas on how to solve this?
Do we need to clear some kind of memory cache?
Do I need a bigger machine with better GPU?
How long should the training take in your tests?
Should I just retrain?

Summary:
Run failed: User program failed with ResourceExhaustedError: OOM when allocating tensor with shape[32,128,52,52] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node conv2d_12/convolution}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [[{{node Mean_1}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Detailed:
Session ID: 4e001867-167d-49da-9618-6dcfc4053686
{"error":{"code":"UserError","message":"User program failed with ResourceExhaustedError: OOM when allocating tensor with shape[32,128,52,52] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc\n\t [[{{node conv2d_12/convolution}}]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.\n\n\t [[{{node Mean_1}}]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.\n","detailsUri":"https://aka.ms/azureml-known-errors","target":null,"details":[],"innerError":null,"debugInfo":{"type":"ResourceExhaustedError","message":"OOM when allocating tensor with shape[32,128,52,52] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc\n\t [[{{node conv2d_12/convolution}}]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.\n\n\t [[{{node Mean_1}}]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.\n","stackTrace":" File "/mnt/batch/tasks/shared/LS_root/jobs/aml-cviotedge-prod/azureml/yolov3_1576506500_05614ed0/mounts/workspaceblobstore/azureml/yolov3_1576506500_05614ed0/azureml-setup/context_manager_injector.py", line 115, in execute_with_context\n runpy.run_path(sys.argv[0], globals(), run_name="main")\n File "/azureml-envs/azureml_6536bbd782c8c80c6934351febe412f9/lib/python3.6/runpy.py", line 263, in run_path\n pkg_name=pkg_name, script_name=fname)\n File "/azureml-envs/azureml_6536bbd782c8c80c6934351febe412f9/lib/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/azureml-envs/azureml_6536bbd782c8c80c6934351febe412f9/lib/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "train.py", line 237, in \n _main(FLAGS.model, FLAGS.fine_tune_epochs, FLAGS.unfrozen_epochs, FLAGS.learning_rate)\n File "train.py", line 110, in _main\n callbacks=[logging, checkpoint, reduce_lr, early_stopping, lossHistory])\n File "/azureml-envs/azureml_6536bbd782c8c80c6934351febe412f9/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper\n return func(*args, **kwargs)\n File "/azureml-envs/azureml_6536bbd782c8c80c6934351febe412f9/lib/python3.6/site-packages/keras/engine/training.py", line 1732, in fit_generator\n initial_epoch=initial_epoch)\n File "/azureml-envs/azureml_6536bbd782c8c80c6934351febe412f9/lib/python3.6/site-packages/keras/engine/training_generator.py", line 220, in fit_generator\n reset_metrics=False)\n File "/azureml-envs/azureml_6536bbd782c8c80c6934351febe412f9/lib/python3.6/site-packages/keras/engine/training.py", line 1514, in train_on_batch\n outputs = self.train_function(ins)\n File "/azureml-envs/azureml_6536bbd782c8c80c6934351febe412f9/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3076, in call\n run_metadata=self.run_metadata)\n File "/azureml-envs/azureml_6536bbd782c8c80c6934351febe412f9/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1439, in call\n run_metadata_ptr)\n File "/azureml-envs/azureml_6536bbd782c8c80c6934351febe412f9/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in exit\n c_api.TF_GetCode(self.status.status))\n","innerException":null,"data":null,"errorResponse":null}},"correlation":null,"environment":null,"location":null,"time":"0001-01-01T00:00:00+00:00"}