Author	Dave Glover, Microsoft Australia
Solution	Azure Machine Learning, Image Classification & Intelligent Edge Devices
Platform	Azure IoT Edge
Documentation	Azure IoT Edge, Azure Custom Vision, Azure Speech services, Azure Functions on Edge, Stream Analytics, Machine Learning services
Video Training	Enable edge intelligence with Azure IoT Edge
Date	As at Oct 2018

• 1. Image Classification with Azure IoT Edge
  • 1.1. Solution Overview
• 2. What is Azure IoT Edge?
  • 2.1. Azure IoT Edge in Action
  • 2.2. Solution Architectural Considerations
• 3. Azure Services
  • 3.1. Creating the Fruit Classification Model
  • 3.2. Exporting an Azure Custom Vision Model
    • 3.2.1. To export your model
  • 3.3. Azure Speech Services
• 4. How to install and run the solution

1. Image Classification with Azure IoT Edge

The scenarios for this Machine Learning Image Classification solution include self-service shopping for vision impaired people or someone new to a country who is unfamiliar with local product names.

1.1. Solution Overview

At a high level, the solution takes a photo of a piece of fruit, gets the name of the fruit from a trained image classifier, converts the name of the fruit to speech and plays back the name of the fruit on the attached speaker.

The solution runs of Azure IoT Edge and consists of a number of services.

1. The Camera Capture Module is responsible for capturing an image from the camera, calling the Image Classification REST API, then calling the Text to Speech REST API and finally playing bask in the classified image label on the speaker.

2. The Image Classification Module is responsible for classifying the image that was passed to it from the camera capture module.

3. The Text to Speech Module passes the text label return from the image classifier module and converts to speech using the Azure Speech Service. As an optimization, this module also caches speech data.

4. USB Camera for Image Capture is used for image capture.

5. A Speaker for text to Speech playback.

6. I used the free tier of Azure IoT Hub for managing, deploying and reporting the IoT Edge device.

7. The Azure Speech to Text service free tier was used for text to speech services.

8. And Azure Custom Vision was used to build the Image Classification model that forms the basis of the Image Classification module.

2. What is Azure IoT Edge?

The solution is built on Azure IoT Edge which is part of the Azure IoT Hub service and is used to define, secure and deploy a solution to an edge device. It also provides cloud-based central monitoring and reporting of the edge device.

The main components for an IoT Edge solution are:-

1. The IoT Edge Runtime which is installed on the local edge device and consists of two main components. The IoT Edge "hub", responsible for communications, and the IoT Edge "agent", responsible for running and monitoring modules on the edge device.

2. Modules. Modules are the unit of deployment. Modules are docker images pulled from a registry such as the Azure Container Registry, or Docker Hub. Modules can be custom developed, built as Azure Functions, or exported services from Azure Custom Vision, Azure Machine Learning, or Azure Stream Analytics.

3. Routes. Routes define message paths between modules and with IoT Hub.

4. Properties. You can set the "desired" properties for a module from Azure IoT Hub. For example, you might want to set a threshold property for a temperature alert.

5. Create Options. Create Options tell Docker runtime what options to start the Module/Docker Container with. For example, you may wish to open ports for REST APIs or debugging ports, define paths to devices such as a USB Camera, set environment variables, or enable privilege mode for certain hardware operations.

6. Deployment Manifest. The Deployment Manifest tells the IoT Edge runtime what modules to deploy and what container registry to pull them from and includes the routes and create options information.

2.1. Azure IoT Edge in Action

2.2. Solution Architectural Considerations

So with that overview of Azure IoT Edge here were my initial considerations and constraints for the solution.

1. The solution should scale from a Raspberry Pi (running Raspbian Linux) on ARM32v7, to my desktop development environment, to an industrial capable IoT Edge device such as those found in the Certified IoT Edge Catalog.

2. The solution required camera input, I used a USB Webcam for image capture as it was supported across all target devices.

3. The camera capture module required Docker USB device pass-through (not supported by Docker on Windows) so that plus targeting Raspberry Pi meant that I need to target Azure IoT Edge on Linux.

4. I wanted my developer experience to mirror the devices I was targeting plus I needed Docker support for the USB webcam, so I developed the solution from Ubuntu 18.04. See my Ubuntu for Azure Developers guide.

   • As a workaround, if your development device is locked to Windows then use Ubuntu in Virtual Box which allows USB device pass-through which you can then pass-through to Docker in the Virtual Machine. A bit convoluted but it does work.

3. Azure Services

3.1. Creating the Fruit Classification Model

The Azure Custom Vision service is a simple way to create an image classification machine learning model without having to be a data science or machine learning expert. You simply upload multiple collections of labeled images. For example, you upload a collection of bananas images and you label them as 'bananas'.

To create your own classification model read How to build a classifier with Custom Vision for more information. It is important to have a good variety of labeled images so be sure to read How to improve your classifier for more information.

3.2. Exporting an Azure Custom Vision Model

This "Image Classification" module in this sample includes a simple fruit classification model that was exported from Azure Custom Vision. Be sure to read how to Export your model for use with mobile devices. It is important to select one of the "compact" domains from the project settings page otherwise you will not be able to export the model.

3.2.1. To export your model

1. From the Performance tab of your Custom Vision project click Export.

   

2. Select Dockerfile from the list of available options

   

3. Then select the Linux version of the Dockerfile.

   

4. Download the docker file and unzip and you have a ready-made Docker solution containing a Python Flask REST API. This was how I created the Azure IoT Edge Image Classification module. Too easy:)

3.3. Azure Speech Services

4. How to install and run the solution

1. Clone this GitHub

2. Deploy your first IoT Edge module to a Linux x64 device

3. Install Visual Studio Code along with IoT Edge Extension

4. Create