goberoi / cloudy_vision

Cloudy Vision is an open source tool to test the image labeling capabilities of different computer vision API vendors.

Cloudy Vision: compare computer vision APIs

Run a corpus of images through multiple computer vision API vendors. View image labeing results side by side so that you can get a general feel for how well each vendor works for your use case. Supported vendors: Microsoft, IBM, Google, Cloudsight, Amazon (Rekognition) and Clarifai.

View example output.

Read this blog post for details.

How it works

1. For a given directory of images, and list of vendors:
2. Call each vendor API for that image (or skip it if cached), e.g. call Google for dog_at_the_park.jpg.
3. Store results in a JSON file with the name: filename.vendor.json, e.g. dog_at_the_park.jpg.google.json.
4. Optionally match the tags returned with your desired tags to test accuracy.
5. Calculate stats around response times, number of tags returned, etc.
6. Create a scaled copy of the original image with height 200px.
7. Generate output.html to show all the images and labeling results in an easy to consume manner.

Usage

1. Get keys for each vendor, and put them in a file called api_keys.json (copy example_api_keys.json to get started).
2. Install dependencies (see below).
3. Place all your images in ./input_images.
4. Run the script: python cloudy_vision.py
5. View ./output/output.html to see results.
6. If you add more images, simply re-run the script. Prior results are cached.

Note for Amazon Rekognition

The keys should not be placed in the api_keys.json file but in ~/.aws/credentials and ~/.aws/config. See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files

Note for Clarifai

The keys should not be placed in the api_keys.json file but in ~/.clarifai/config. See https://github.com/Clarifai/clarifai-python#setup

Desired Tags

You can specify tags that you hope to get, and see whether results from each vendor match. We'll compute these additional stats:

• matching_tags_count (average and standard deviation): number of api tags matching the image tags
• matching confidence (average and standard deviation): confidence of the matching tags

To work with tagged images, set the tagged_images setting to True and fill a tags.json file (copy example_tags.json to get started; this file contains a map image_filename => tags).

Installation

Install these dependencies:

virtualenv venv
source venv/bin/activate
pip install -r requirements.txt

Note that installing Pillow can be dropped if you set settings['resize']=False.

Contributing

If you make modifications that may help others, please fork and send me a pull request. Some ideas for contributions: (a) add new image recognition vendors, (b) expose more attributes per vendor, e.g. face detection, (c) bugs, requests, feedback.

Credits

Authored by @goberoi.

Thanks to @lucasdchamps for several features: response time stats; matching with desired tags; Amazon Rekognition; and several other improvements.

License

MIT License - Copyright (c) 2016 Gaurav Oberoi.

cloudy_vision's Issues

Amazon

Is it hard to add another vendor Amazon Rekognition to your cloudy_vision project?
https://aws.amazon.com/rekognition/

cloudsight.errors.APIError: Invalid byte sequence - Cloudsight API

I'm getting an error using Cloudsight API, on windows.

Traceback (most recent call last):
  File "cloudy_vision.py", line 264, in <module>
    process_all_images()
  File "cloudy_vision.py", line 198, in process_all_images
    api_result = vendor_module.call_vision_api(filepath, settings('api_keys'))
  File "C:\Users\EdanWeis\Documents\cloudy_vision\vendors\cloudsight_.py", line 14, in call_vision_api
    response = api.image_request(image_file, image_filename)
  File "C:\Python27\lib\site-packages\cloudsight\api.py", line 114, in image_request
    return self._unwrap_error(response)
  File "C:\Python27\lib\site-packages\cloudsight\api.py", line 88, in _unwrap_error
    raise errors.APIError(json_response['error'])
cloudsight.errors.APIError: Invalid byte sequence

Any suggestions? I've contacted Cloudsight for comment.

Excellent Review of VISION APIs

I was not aware of CloudSight's API... I liked it because it gives you a complete phrase and not chunks of words for the description. Although it looks costly at least for my budget.

On the other hand, since you have worked parsing JSON from Microsoft, I wonder if you could kindly give me some idea of how to get an specific value from a JSON array, in this case from Microsoft OCR API... I am including the link of the JSON "results dump" I got from their server, but I am having a really hard time... simply getting the TEXT result values (which are the words being turned into editable text) and then concatenate them into a single javascript value.

LINK:

http://mycloudvip.com/azure/js/ocr-test-values.js

I would greatly appreciate your thoughts on this topic.

Thanks

Hugo Barbosa