hollance / mobilenet-coreml

The MobileNet neural network using Apple's new CoreML framework

MobileNet with CoreML

This is the MobileNet neural network architecture from the paper MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications implemented using Apple's shiny new CoreML framework.

This uses the pretrained weights from shicai/MobileNet-Caffe.

There are two demo apps included:

• Cat Demo. Shows the prediction for a cat picture. Open the project in Xcode 9 and run it on a device with iOS 11 or on the simulator.

• Camera Demo. Runs from a live video feed and performs a prediction as often as it can manage. (You'll need to run this app on a device, it won't work in the simulator.)

Note: Also check out Forge, my neural net library for iOS 10 that comes with a version of MobileNet implemented in Metal.

Converting the weights

The repo already includes a fully-baked MobileNet.mlmodel, so you don't have to follow the steps in this section. However, in case you're curious, here's how I converted the original Caffe model into this .mlmodel file:

1. Download the caffemodel file from shicai/MobileNet-Caffe into the top-level folder for this project.

Note: You don't have to download mobilenet_deploy.prototxt. There's already one included in this repo. (I added a Softmax layer at the end, which is missing from the original.)

2. From a Terminal, do the following:

$ virtualenv -p /usr/bin/python2.7 env
$ source env/bin/activate
$ pip install tensorflow
$ pip install keras==1.2.2
$ pip install coremltools

It's important that you set up the virtual environment using /usr/bin/python2.7. If you use another version of Python, the conversion script will crash with Fatal Python error: PyThreadState_Get: no current thread. You also need to use Keras 1.2.2 and not the newer 2.0.

3. Run the coreml.py script to do the conversion:

$ python coreml.py

This creates the MobileNet.mlmodel file.

4. Clean up by deactivating the virtualenv:

$ deactivate

Done!

mobilenet-coreml's Issues

How long does it take to predict an image

Thanks for your work.I think I run your demo successfully now.But I want to know how long does it take to predict an image so I add some code to test the time. And when I loop for 100 times for predict the cat image, it cost about 20s to do this on my iPhone6, But I think it should be faster.So am I wrong?

build succeed but crash

Model not working for me

In both the Cat and Camera demos, the model only predicts "candle, taper, wax light", and at 100% or very close to 100% confidence. I've tried both the provided model and ran the trainer myself with the same results. Any idea what I could be doing wrong?

[Question] How does MobileNet.mlmodel compare to VGG16.mlmodel

Hi! Nice work!
I'm interested in how this MobileNet.mlmodel compares to the ones provided by Apple on their download page. Specifically how it compares to the VGG16 model which I've been using.

Generally I'm on a quest to find the best (biggest) object classification model to use in my app.
Maybe you have some useful suggestions?

Anyway, thanks for providing this sample project!

Getting the output from intermediate layers in the mobilenetv2 network

@hollance

I am trying to get the intermediate layer(add_node) output and merge it into the existing model outputs (confidences and coordinates)

Experiments :

• Was able to get the add output from the ssd model, also passed this output as an input to decode model stage where i just add a dummy permute node.
• In the NMS model, it has a fixed set of inputs and outputs set by (confidenceInputFeatureName, confidenceOutputFeatureName ..etc) didn't find anything to forcefully create a new input as its set by the NMS_suppression.proto file
• So thought of creating a new model part that takes input as the NMS outputs(confidences and coordinates) and the previous decode layer add_output.

with this the coreml model is created as attached below.

But while loading it on through python it gives

an error saying:
RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "Error reading protobuf spec. validator error: Pipeline: Input 'confidence' of model 'CoreML.Specification.ModelDescription' does not match the type previously specified by the pipeline input or the output of a previous model.".

I am assuming this error is prompted bcoz the new model created is taking the input, not from the intermediate layer. Can we add a dummy node in the NMS model to bypass this ... any thoughts on this will be helpful or even is this the correct way of doing it.

I have attached the coreml and convert pythod code used

Archive.zip

Where is synset_words.txt

Could not find the synset_words.txt file.

[Question] Where are the conversion parameters come from?

Hi! This project helped me a lot, thank you!

I'm new to machine learning and I have a question about coreml.py.
It specifies some parameters to the convert function.
Where are these parameters come from? What do these actual values (like 0.017) mean?
Especially, scale, is_bgr, red_bias, green_bias and blue_bias.
It will be very helpful if anyone could give me some information about it.
Thank you✨

scale = 0.017

coreml_model = coremltools.converters.caffe.convert(
    ('mobilenet.caffemodel', 'mobilenet_deploy.prototxt'),
    image_input_names='data',
    is_bgr=True, image_scale=scale,
    red_bias=-123.68*scale, green_bias=-116.78*scale, blue_bias=-103.94*scale,
    class_labels='synset_words.txt')

Model looks unoptimised

1. BN can be fused
2. batchnorm and scale are separated layers here this is because of bad Caffe implementation.

BTW is there any app where we just can benchmarks some other pertained classification models on iPhone?

Get wrong result after converting the model to coreML

Hello! I have used the caffe model you provided to get a semantic segmentation(two classes) model (just change to model into a FCN model). Then I convert the caffe model to coreML model. I can get reasonable result on iphone X but the result is not so good that is different from the result I get from caffe model on Ubuntu. Have you met problems like this? Is there any difference between caffe and coreML that matters? I am confused with it and thanks for you reply.

Error: When trying to convert Caffe model to CoreML model

Hi, Thanks for this excellent repo. But when I am getting the following error when trying to convert the caffe model to .mlmodel.

Traceback (most recent call last):
  File "coreml.py", line 23, in <module>
    class_labels='synset_words.txt')
  File "//anaconda/envs/coreml/lib/python2.7/site-packages/coremltools/converters/caffe/_caffe_converter.py", line 131, in convert
    return MLModel(model_path)
  File "//anaconda/envs/coreml/lib/python2.7/site-packages/coremltools/models/model.py", line 126, in __init__
    self.__proxy__ = _get_proxy_from_spec(model)
  File "//anaconda/envs/coreml/lib/python2.7/site-packages/coremltools/models/model.py", line 55, in _get_proxy_from_spec
    return _MLModelProxy.fromSpec(filename)
RuntimeError: Got non-zero exit code 72 from xcrun. Output was: xcrun: error: unable to find utility "coremlcompiler", not a developer tool or in PATH

I tried checking online but no help. Can you please guide me related to this problem?

Why is Keras necessary?

Could you explain why it's necessary to download Keras? It looks like all you are doing is using coreml's caffe converter, which I'm under the impression doesn't rely on Keras in any way.