khesualdo / ms-azure-ai-hackathon-s19

:cherries: :lemon: :watermelon: FRÜT utilizes the Azure Cognitive Vision API to assist Border Service Officers in identifying different fruits as well as provides information associated with the identified fruit.

FRÜT 🍒 🍋 🍉

🍒 🍋 🍉 FRÜT utilizes Azure Cognitive Vision API to assist Border Service Officers in identifying different fruits as well as provides information associated with the identified fruit.

Inspiration

The inspiration comes from this video. In the video it shows how hard the Border Service Officers work to protect the American wild life from illegal imports of produce.

While, it is easy to spot rotten fruits, if the fruits are in perfect condition, the agent may or may not know if the fruit is allowed into the country. Also, it is definitely easy to identify common fruits, such as bananas or oranges, however, it is more difficult to identify more exotic fruits, such as cherimoyas, sweetsops, etc... It gets worse when fruits look very similar, for example - strawberry guavas and cranberries, lima and (large) limes or even unripe oranges.

Strawberry guavas on the left / Cranberries on the right

Limas on the left / Limes on the right

An agent may make a mistake by confiscating permitted items, in this case the people are negatively affected by having to pay a fine. On the other hand, an agent may make a mistake by not confiscating non-permitted items, in this case the entire country is negatively affected by having to pay an even bigger "fine" to treat unknown insects and plant diseases.

About

FRÜT could assist Border Service Officers in identifing different fruits, along with their attributes - such as fruit type, history, origin, as well as whether the fruit is allowed into the country.

FRÜT could also be used by regular citizens during their shopping trips.

Implementation

Screenshots of the application: 1, 2

Run

Create a Function App resource on Azure portal

• Select HTTP trigger as type
• Copy code from function.csx
• Set FUNCTION_URL in index.js

Go to Custom Vision Portal and create a new project

• Set Project Type as Classification
• Set Classification Types as Multiclass
• Set Domains as Food
• Set COGNITIVE_URL in index.js for image file (available after publishing the project)
• Set COGNITIVE_KEY in index.js (available after publishing the project)

If you get the We only support publishing to a prediction resource in the same region as the training resource the project resides in. error, create a Custom Vision resource. Make sure it is in the same region as all of your other resources. Tutorial on how to use the Custom Vision Portal.

Run Client

> cd client/
> http-server -c-1

http-server on npm

Run Server

> cd server/
> node index.js

Challenges

Surprisingly the biggest challenge was capturing and then sending the image to the back-end. Working so closely with Base64 Encoding, Data URIs, as well as, byte arrays was new for me. I am glad I figured it out in the end!

Technologies

• Azure Cognitive Services (Custom Vision API)
• Azure Functions
• HTML, CSS, JS
• NodeJS, ExpressJS
• Fruits 360 Dataset