microsoft / az-ml-realtime-score

Architecture for deploying real-time scoring of machine learning models using Azure Machine Learning

Authors: Fidan Boylu Uz, Yan Zhang, Mario Bourgoin

Acknowledgements: Mathew Salvaris

Deploying Python models for real-time scoring using Azure Machine Learning

In this repository there are a number of tutorials in Jupyter notebooks that have step-by-step instructions on (1) how to train a machine learning model using Python; (2) how to deploy a trained machine learning model throught Azure Machine Learning (AzureML). The tutorials cover how to deploy models on following deployment target:

Overview

This scenario shows how to deploy a Frequently Asked Questions (FAQ) matching model as a web service to provide predictions for user questions. For this scenario, “Input Data” in the architecture diagram refers to text strings containing the user questions to match with a list of FAQs. The scenario is designed for the Scikit-Learn machine learning library for Python but can be generalized to any scenario that uses Python models to make real-time predictions.

Design

The scenario uses a subset of Stack Overflow question data which includes original questions tagged as JavaScript, their duplicate questions, and their answers. It trains a Scikit-Learn pipeline to predict the match probability of a duplicate question with each of the original questions. These predictions are made in real time using a REST API endpoint. The application flow for this architecture is as follows:

1. The client sends a HTTP POST request with the encoded question data.
2. The webservice extracts the question from the request
3. The question is then sent to the Scikit-learn pipeline model for featurization and scoring.
4. The matching FAQ questions with their scores are then piped into a JSON object and returned to the client.

An example app that consumes the results is included with the scenario.

Prerequisites

1. Linux (Ubuntu).
2. Anaconda Python
3. Docker installed.
4. Azure account.

---

NOTE You will need to be able to run docker commands without sudo to run this tutorial. Use the following commands to do this.

sudo usermod -aG docker $USER
newgrp docker

---

The tutorial was developed on an Azure Ubuntu DSVM, which addresses the first three prerequisites.

Setup

To set up your environment to run these notebooks, please follow these steps. They setup the notebooks to use Azure seamlessly.

1. Create a Linux Ubuntu VM.
2. Log in to your VM. We recommend that you use a graphical client such as X2Go to access your VM. The remaining steps are to be done on the VM.
3. Open a terminal emulator.
4. Clone, fork, or download the zip file for this repository:

   git clone https://github.com/Microsoft/az-ml-realtime-score.git
   

5. Enter the local repository:

   cd az-ml-realtime-score
   

6. Copy sample_workspace_conf.yml to a new file, workspace_conf.yml, and fill in each field. This will keep secrets out of the source code, and this file will be ignored by git.
7. Create the Python az-ml-realtime-score virtual environment using the environment.yml:

   conda env create -f environment.yml
   

8. Activate the virtual environment:

   source activate az-ml-realtime-score
   

   The remaining steps should be done in this virtual environment.
9. Login to Azure:

   az login
   

   You can verify that you are logged in to your subscription by executing the command:

   az account show -o table
   

10. Start the Jupyter notebook server:

    jupyter notebook
    

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repositories using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Related projects

Microsoft AI Github Find other Best Practice projects, and Azure AI Designed patterns in our central repository.

az-ml-realtime-score's Issues

blocked on edge device creation

I am blocked when trying to create the edge device. This is the error message i get

Azure Error: MarketplacePurchaseEligibilityFailed
Message: Marketplace purchase eligibilty check returned errors. See inner errors for details. 
Exception Details:
        Error Code: BadRequest
        Message: Offer with PublisherId: microsoft_iot_edge, OfferId: iot_edge_vm_ubuntu cannot be purchased due to validation errors. See details for more information.[{"Legal terms have not been accepted for this item on this subscription. To accept legal terms using PowerShell, please use Get-AzureRmMarketplaceTerms and Set-AzureRmMarketplaceTerms API(https://go.microsoft.com/fwlink/?linkid=862451) or deploy via the Azure portal to accept the terms":"StoreApi"}]

When image creation fails, the test coninues without reporting error right after it

When using Makefile to do test all notebooks together, if the image creation failed in notebook 04_CreateImage.ipynb, the run does not stop at the cell where the failure happens. It continues to run through the remaining cells in notebook 04_CreateImage.ipynb. Then it continues to run the next notebook (05_DeployOnAKS.ipynb) and stop in this notebook reporting cannot create web service from failed image.

Ideally the run should stop at the cell when the image creation fails.

programatic creation and management of iot hub

In notebook 5, 05_deployoniotedge, instead of sending users to a documentation page, we should use programatically deploy our iothub and register device (perhaps using the CLI)

Running notebooks takes a long time in DevOps pipeline

The notebooks take a long time to run in DevOps pipeline. This is because the compute available is limited. To speed things up we should parameterise the scripts to use less compute resources.

Cookiecutter / seeing notebooks in the repo

The one major disadvantage of using cookie cutter is that this happy path becomes less usable for people who want to just open notebooks directly in Github. Users would have to know to open the cookie cutter dir, and follow the notebooks until 04, at which point they'll have to know to open either the aks deployment folder or the iot folder to continue seeing the rest of the notebooks.

As a way around this, perhaps we can include instructions in the readme to guide people through that - assume that at least a pretty large % of the users are just coming to the Github page, and learning from the notebooks directly on Github.

Maybe there is some clever way to do symlinks as well?

Add Build Badge to Readme file

At the top of the readme, add the build badge:

https://dev.azure.com/customai/MLAKSDeployAMLPipeline/_apis/build/status/Microsoft.MLAKSDeployAML?branchName=master

This will reflect the status of the last/latest build of the code.

Print out connection string in setup to help instrumentation debugging

iotedge architecture description

It might be worth adding a description, either in the README of the IOT Edge folder or at the top of notebook 05 (05_deployoniotedge), that talks about how the iot edge deployment works. What does the architecture look like? What is the 'edge device' in this tutorial? etc

Rename itemSelector to item_selector

04_create_image permission error

WebserviceException: Failed to create Docker client: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

Getting this error in the last cell of the 04 notebook. I haven't dug into what may be causing it, but it seems to break here just running the notebooks as is.

We may need to restrict tornado<6 in environment.yaml

jupyter/notebook#4311

Repo name is a bit misleading for iotedge scenario

import AuthenticationException missing in MLAKSDeployAML/utilities.py

I think this si missing:
from azureml.core.authentication import AuthenticationException

notebook 03 references AKS even when selecting IOTEdge scenario

last cell in 03_DevelopScoringScript: Next, we move on to [creating the docker image and deploying on AKS](04_CreateImage.ipynb).

notebooks in 00 & 04 (under iotedge) on tags view

Harden core validation

Core check in the aks/05 notebook is too generic. There is a potential to get the wrong SKU quota and make assumptions (good or bad) based on that.

Missing image Design.png

Split script create_model.py into functions

Replace throughput test with locust

see https://github.com/Microsoft/Recommenders/blob/gramhagen/load_test/notebooks/05_operationalize/aks_locust_load_test.ipynb

No link to next notebook at the bottom of 04_create_image notebook

Can a link be created depending on whether the user selects the AKS or the IOT deployment?