microsoft / azure-percept-advanced-development

Azure Percept DK advanced topics

Azure Percept DK Advanced Development

Azure Percept DK and the associated Advanced Developer experience for the Percept DK will be retired March 30th, 2023. Click here to learn more: Azure Percept DK Retirement Announcement | Azure updates | Microsoft Azure

Releases

All the releases are tagged with release notes here.

Overview

This repository holds all the code and documentation for advanced development using the Azure Percept DK. In this repository, you will find:

• azureeyemodule: The code for the azureeyemodule, which is the IoT module responsible for running the AI workload on the Percept DK.
• machine-learning-notebooks: Example Python notebooks which show how to train up a few example neural networks from scratch (or using transfer learning) and get them onto your device.
• Model and Data Protection: Azure Percept currently supports AI model and data protection as a preview feature.
• Bring Your Own Model Pipeline Tutorials: Tutorials for how to bring your own custom AI model (and post-processing pipeline) to the device.

General Workflow

One of the main things that this repository can be used for is to bring your own custom computer vision pipeline to your Azure Percept DK. The flow for doing that would be this:

1. Use whatever version of whatever DL framework you want (Tensorflow 2.x or 1.x, PyTorch, etc.) to develop your model.
2. Save it to a format that can be converted to OpenVINO IR. However, make sure your ops/layers are supported by OpenVINO 2021.1. See here for a compatiblity matrix.
3. Use OpenVINO to convert it to IR or blob format.
   • I recommend using the OpenVINO Workbench to convert your model to OpenVINO IR (or to download a common, pretrained model from their model zoo).
   • You can use the scripts/run_workbench.sh script on Unix systems to run the workbench, or scripts/run_workbench.ps1 script on Windows systems.
   • You can use a Docker container to convert IR to blob for our device. See the scripts/compile_ir.sh script and use it as a reference (for Windows users: scripts/compile_ir.ps1). Note that you will need to modify it to adjust for if you have multiple output layers in your network.
4. Develop a C++ subclass, using the examples we already have. The Azure Percept DK's azureeyemodule runs a C++ application which needs to know about your model in order for your model to run on the device. See the tutorials and azureeyemodule folder for how to do this.
5. The azureeyemodule is the IoT module running on the device responsible for doing inference. It will need to grab your model somehow. For development, you could package your model up with your custom azureeyemodule and then have the custom program run it directly. You could also have it pull down a model through the module twin (again, see the azureeyemodule folder and tutorials for more details).

See the tutorials for detailed walkthroughs on how to do all of this.

Note the following limitations while developing your own custom model for the device:

• Only certain ops/layers are supported.
• Model inputs and outputs must be either unsigned 8 bit integers or 32-bit floating point values - other values are not currently supported by the OpenCV G-API.
• The model should be small. This is an embedded device for doing inference at the very edge of the IoT network. Its usual use case would be to help inform a larger downstream model or to make simple decisions, like whether something moving is a cat or a person - not to do instance segmentation over 200 different classes. In particular, while there is no hard and fast limit to model size, I have personally found that models larger than 100 MB (after conversion to OpenVINO .blob format) have serious trouble getting loaded onto the device and running. This is just a rule of thumb though; the point is to keep your models small.
• The hardware accelerator on the device is an Intel Myriad X. It does not support quantization - only FP32 and FP16 are allowed weight values.
• The VPU is best suited for accelerating convolutional architectures. Having said that, recurrent architectures are supported (as long as your ops are in the compatibility matrix), they just aren't very fast. The Optical Character Recognition example model actually makes use of an RNN.
• You are not technically limited to a single model. You can bring cascaded models and stitch them into the G-API, but keep in mind that this will typically result in slower inferences, especially if you are doing any processing of the outputs from one network before feeding into the next, as these intermediate values will need to be offloaded from the VPU back to the ARM chip for computation. For an example of how to do this, see the OCR example model file.

Model URLs

The Azure Percept DK's azureeeyemodule supports a few AI models out of the box. The default model that runs is Single Shot Detector (SSD), trained for general object detection on the COCO dataset. But there are a few others that can run without any hassle. Here are the links for the models that we officially guarantee (because we host them and test them on every release).

To use these models, you can download them through the Azure Percept Studio, or you can paste the URLs into your Module Twin as the value for "ModelZipUrl". Not all models are found in the Azure Percept Studio.

Model	Source	License	URL
Faster RCNN ResNet 50	Intel Open Model Zoo	Apache 2.0	https://aedsamples.blob.core.windows.net/vision/aeddevkitnew/faster-rcnn-resnet50.zip
Open Pose	Intel Open Model Zoo	Apache 2.0	https://aedsamples.blob.core.windows.net/vision/aeddevkitnew/openpose.zip
Optical Character Recognition	Intel Open Model Zoo and Intel Open Model Zoo	Apache 2.0	https://aedsamples.blob.core.windows.net/vision/aeddevkitnew/ocr.zip
Person Detection	Intel Open Model Zoo	Apache 2.0	https://aedsamples.blob.core.windows.net/vision/aeddevkitnew/person-detection-retail-0013.zip
Product Detection	Custom Vision	Apache 2.0	https://aedsamples.blob.core.windows.net/vision/aeddevkitnew/product-detection.zip
SSD General	Intel Open Model Zoo	Apache 2.0	https://aedsamples.blob.core.windows.net/vision/aeddevkitnew/ssdlite-mobilenet-v2.zip
Tiny YOLOv2 General	Intel Open Model Zoo	Apache 2.0	https://aedsamples.blob.core.windows.net/vision/aeddevkitnew/tiny-yolo-v2.zip
Unet for Semantic Segmentation of Bananas (for this notebook)	Trained from scratch	GPLv3	https://aedsamples.blob.core.windows.net/vision/aeddevkitnew/binary-unet.zip
Vehicle Detection	Custom Vision	Apache 2.0	https://aedsamples.blob.core.windows.net/vision/aeddevkitnew/vehicle-detection.zip

Contributing

This repository follows the Microsoft Code of Conduct.

Please see the CONTRIBUTING.md file for instructions on how to contribute to this repository.

Trademark Notice

Trademarks This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

Reporting Security Vulnerabilities

Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include Microsoft, Azure, DotNet, AspNet, Xamarin, and our GitHub organizations.

If you believe you have found a security vulnerability in any Microsoft-owned repository that meets Microsoft's Microsoft's definition of a security vulnerability, please report it to us as described below.

Please do not report security vulnerabilities through public GitHub issues.

Instead, please report them to the Microsoft Security Response Center (MSRC) at https://msrc.microsoft.com/create-report.

If you prefer to submit without logging in, send email to [email protected]. If possible, encrypt your message with our PGP key; please download it from the Microsoft Security Response Center PGP Key page.

You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at microsoft.com/msrc.

Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:

• Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
• Full paths of source file(s) related to the manifestation of the issue
• The location of the affected source code (tag/branch/commit or direct URL)
• Any special configuration required to reproduce the issue
• Step-by-step instructions to reproduce the issue
• Proof-of-concept or exploit code (if possible)
• Impact of the issue, including how an attacker might exploit the issue

This information will help us triage your report more quickly.

If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our Microsoft Bug Bounty Program page for more details about our active programs.

We prefer all communications to be in English.

Microsoft follows the principle of Coordinated Vulnerability Disclosure.

azure-percept-advanced-development's Issues

Provide the WebStreamModule code

It would be nice if you could provide the WebStreamModule sample code.

There are multiple links in your Readme that lead to a 404 error [BUG]

format that can be converted to OpenVINO IR.

here for a compatiblity matrix.

Only certain ops/layers are supported.

There may be more, but these were ones I needed more info on. Not sure, perhaps the OpenVINO site is down temporarily, and this is only temporary... Worth testing.

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

[Need Help] myriad_compile step is failing in my environemt

Describe the bug
Not sure if its a bug, my environment is not the same.
Repeated error message below when doing the myriad_compile step
E: [xLinkUsb] [ 367214] [myriad_compile] usb_find_device_with_bcd:266 Library has not been initialized when loaded

To Reproduce
Run up to Step 9 of https://github.com/microsoft/azure-percept-advanced-development/blob/main/machine-learning-notebooks/train-from-scratch/SemanticSegmentationUNet.ipynb
I'm doing the steps locally (not on a Azure ML remote compute instance). I have the model built locally.
Next, I was taking the model thru the conversion steps in the openvino container.
Container is openvino/ubuntu18_dev:2021.3
Prior steps, listed below, succeeded:

• pytorch model to onnx - OK
• onnx to IR - OK
• IR to blob - Fail

The exact command is

source /opt/intel/openvino_2021/bin/setupvars.sh 

/opt/intel/openvino_2021/deployment_tools/inference_engine/lib/intel64/myriad_compile \                                                                                                                    
8    -m intel/bananas.xml -o intel/bananas.blob -VPU_NUMBER_OF_SHAVES 8 -VPU_NUMBER_OF_CMX_SLICES 8 -ip U8 -op FP32 

Thank you for taking a look.

Update: seems harmless as it generated a blob file.

[BUG]

Step 1 of the tutorial fails.

It states "create a new folder in your [AML] workspace"

1. there is nothing called "workspace"
   but I do see a Azure Machine Learning (AML) Studio

2. There is no 'folder' option in the Create New menu

3. There really should be a link in step one to assist new developers to the correct 'workspace'.

Essential the entire tutorial is out of reach because step 1 cannot be completed.

[BUG] Couldn't find the default object detection model in OpenVINO

Describe the bug
I tried to find the OpenVINO version of the default object detection model, ssd_mobilenet_v2_coco.blob, but couldn't find it.

In README.md, it said the default SSD model is from here. However, I checked the labels, the model on Percept has 183 labels, but the model in OpenVINO has only 92 labels. Where is the OpenVINO version of the default ssd_mobilenet_v2_coco.blob?

To Reproduce
None

Expected behavior
The labels on Azure Percept should be the same as The labels in OpenVINO.

Screenshots
None

Logs
None

Additional context
None

Question: Will other languages be supported?

Firstly, thank you for great documents.

It seems like currently only C++ is supported to develop edge application.
In a future, will Percept support other language? (e.g. Python, Rust)

Thank you.

PyTorchModelFromScratch training script

Describe the bug

After submitting the training job:
run = experiment.submit(config)

The training script gives an error:
User program failed with HTTPError: HTTP Error 404: Not Found

Driver log attached:
70_driver_log.txt

Add inference script to the deployed model/ access the video frame

Hi, is there a way to add an inference script (like when deploying a model in azure ml) to do per/postprocessing? if not is there a way to access the frames of the camera video stream?

what I want to achieve is to use a pre-trained model deployed in a container to detect an object in an image frame and then do some postprocessing and send the result with the image frame to another deployed container

Tutorial crashing

I'm pretty new to this software, and I have been trying to do the bananas tutorial for a couple days now. I keep getting to the part where we must run the code (which I know takes hours), but it always ends up crashing after 7 hours.

Not sure if this is because I am doing something wrong, or maybe because I am on MacOS, but any help would be appreciated

azure percept point to new iothub

If i create a brand new iothub with different name. how to i get azure percept to work again

i tried editing the device_connection_string in /etc/iotedge/config.yaml
it connects to the device in the new iothub but only has two modules.
edgeAgent and edgeHub

[BUG] RunHistory initialization failed: libffi.so.7: cannot open shared object file: No such file or directory

Describe the bug
While following the steps in Banana Tutorial under the section "Running the notebook" I am seeing this error appear:

"error": {
"code": "ServiceError",
"severity": null,
"message": "AzureMLCompute job failed.\nServiceError: runTaskLetTask failed because: libffi.so.7: cannot open shared object file: No such file or directory\n\tReason: Job failed with non-zero exit Code",
"messageFormat": null,
"messageParameters": null,
"referenceCode": null,
"detailsUri": null,
"target": null,
"details": [],
"innerError": null,
"debugInfo": null,
"additionalInfo": null
},
"correlation": {
"operation": "cf742550df05044dbf2b80b3397f31d7",
"request": "8ae5bc2bda1f4e59"
},
"environment": "australiaeast",
"location": "australiaeast",
"time": "2021-07-16T05:26:14.7993082+00:00",
"componentName": "execution-worker"

} <

The note book I am running is SemanticsSegmentationUNet.ipynb

Screenshots

[QUESTION] - How to interact with the bluetooth module

I am trying to find a way to combine data from the vision module and data from the Bluetooth module. It is quite hard to determine the right approach for using the Bluetooth module. i was hoping to find any guidance on this here??

[BUG] stoi error when running compile_and_test.[sh|ps1] scripts

Describe the bug
Following the PyTorch from Scratch Tutorial I get the following error, when trying to run detection through the mock-eye-module:

terminate called after throwing an instance of 'std::invalid_argument'  
  what():  stoi

The model used is pulled from openvino workbench zoo. The default ssd model to test the mock-eye-module container. Hence error should not be related to the model ssd_mobilenet_v2_coco.

Mock-eye-module by itself is compiling fine.

Based on simple debugging, error seems to be from the G-API when executed.

To Reproduce
Steps to reproduce the behavior:
Follow all steps in Prerequisites.

When converted model, and video person-bicycle-car-detection.mp4 downloaded.

I run the following on mac:

./scripts/compile_and_test.sh --video=test-artifacts/person-bicycle-car-detection.mp4 --weights=test-artifacts/ssd_mobilenet_v2_coco.bin --xml=test-artifacts/ssd_mobilenet_v2_coco.xml

I run the following on windows:

./scripts/compile_and_test.ps1 -ipaddr <your IP address> -xml test-artifacts/ssd_mobilenet_v2_coco.xml -video test-artifacts/person-bicycle-car-detection.mp4

Both return:

[setupvars.sh] OpenVINO environment initialized
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenCV: /opt/intel/openvino_2021.1.110/opencv (found version "4.5.0") found components:  gapi highgui 
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") 
-- Checking for module 'gstreamer-1.0>=1.14'
--   Found gstreamer-1.0, version 1.16.2
-- Found InferenceEngine: /opt/intel/openvino/deployment_tools/inference_engine/lib/intel64/libinference_engine.so (Required is at least version "2.0") 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/openvino/tmp/build
Scanning dependencies of target mock_eye_app
[ 14%] Building CXX object CMakeFiles/mock_eye_app.dir/main.cpp.o
[ 28%] Building CXX object CMakeFiles/mock_eye_app.dir/kernels/ssd_kernels.cpp.o
[ 42%] Building CXX object CMakeFiles/mock_eye_app.dir/kernels/utils.cpp.o
[ 57%] Building CXX object CMakeFiles/mock_eye_app.dir/modules/device.cpp.o
[ 71%] Building CXX object CMakeFiles/mock_eye_app.dir/modules/objectdetection/object_detectors.cpp.o
[ 85%] Building CXX object CMakeFiles/mock_eye_app.dir/modules/parser.cpp.o
[100%] Linking CXX executable mock_eye_app
[100%] Built target mock_eye_app
Cannot open labelfile /home/openvino/tmp/labels.txt
Labels will not be available.
terminate called after throwing an instance of 'std::invalid_argument'
  what():  stoi

Expected behavior
Based on the documentation, a window should pop out and stream a video with object detection overplayed.

Additional context
Additional observation
The scripts compile_and_test.sh expects an "example" folder in the mock-eye-module, not included when cloning the repo.
The "mock-eye-module-debug" pulls "openvino/ubuntu18_runtime:latest" instead of "openvino/ubuntu18_runtime:2021.1".

[FEATURE-REQUEST] Document Enhancement Request

When reading the document, we feel the following aspects are not clear enough. Some points mentioned below are not easy to find:

• An overview page that shows people e2e structure of IoT edge modules running on Azure Percept, including relationships between different modules, providing links to different resources.
• A well-structured documentation that gives instructions to developers and machine learning engineers (or data scientists) on how to add custom modules.
• A clear documentation that shows how to compile azureeyemodule end-to-end on Windows and Linux OS. Current document provided in github is not clear in the following aspects:
  • Differences between native compile and non-native compile?
  • When to use a native compile method and when to use non-native compile method?
  • how to revert (or fixed) if any steps go wrong? (reset?)
• The documentation doesn't point out modules to be stopped if re-deploy custom azureeyemodule is needed -- we need to stop both azureeyemodule and IoT agent
• The document doesn't tell the readers that rtsp:3000 is working when you customize it using a native way.
• The document doesn't state that the readers can check 8554 using a VLC player
• The document should clearly state expected drivers for external devices, for example, whether a driver for external monitor is expected.
• The document should be revisited and updated frequently based on the OpenVINO update, and check if the recommended approaches of converting models to blob are still valid.
• The document should state that it is possible to convert model on device using the built-in code provided by azureeyemodule, along with the required model files and format. Currently, when reading the code, we understand it is possible to do so, but no instructions or sample model files are provided.
• It would be nice if the document provided a reference link to OpenCV G-API
• It would be nice if the document shows the structure of the model zip file required by azureeyemodule.
• It would be nice if the hardware spec could also provide the GFLOPs besides TOPs. Or provide latency benchmark on the default models provided in azureeyemodule repo.

Unable to start tutorial

While attempting the prerequisites, executing "run_workbench.ps1", I get this error.

`.\run_workbench.ps1 : File C:\percept\azure-percept-advanced-development\scripts\run_workbench.ps1 cannot be loaded.
The file C:\percept\azure-percept-advanced-development\scripts\run_workbench.ps1 is not digitally signed. You cannot
run this script on the current system. For more information about running scripts and setting execution policy, see
about_Execution_Policies at https:/go.microsoft.com/fwlink/?LinkID=135170.
At line:1 char:1

• .\run_workbench.ps1

•   + CategoryInfo          : SecurityError: (:) [], PSSecurityException
    + FullyQualifiedErrorId : UnauthorizedAccess`
  
  

How do I digitally sign this script for you?

Audio stream or azureearmodule

Hi,

great news with this development sdk.
However, I was wondering where you document the custom possibilities with the audio module.
Would be interested to get access to the audio stream.
Do you plan to have a similar module as with azureeymodule?

Thanks

[BUG] Code in ssd.cpp result in compiling error

Describe the bug
Code azure-percept-advanced-development/ssd.cpp at main · Microsoft/azure-percept-advanced-development (github.com) results in compile error when considering camera feed.

To Reproduce
recompile the azureeyemodule using the current code

Expected behavior
no compile error

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
We modified the following to make it work:

// before: auto pipeline = graph.compileStreaming(cv::compile_args(networks, kernels, cv::gapi::mx::mvcmdFile{ this->mvcmd })); 
auto pipeline = graph.compileStreaming(cv::gapi::mx::Camera::params(), cv::compile_args(networks, kernels, cv::gapi::mx::mvcmdFile{ this->mvcmd }));
...
// before: pipeline.setSource<cv::gapi::wip::GCaptureSource>(video_file_path);
pipeline.setSource(cv::gapi::wip::make_src<cv::gapi::mx::Camera>()); 
...
// before: pipeline.setSource<cv::gapi::wip::GCaptureSource>(0); 
pipeline.setSource(cv::gapi::wip::make_src<cv::gapi::mx::Camera>());

[BUG] azureeyemodule can not access uvc camera at 2112-1 version

Seem azureeyemodule:2112-1 uses non-root permission and access uvc camera will get error.
Any suggestion for accessing uvc camera at azureeyemodule:2112-1?

Thank you.