dailystudio / ml Goto Github PK

is there any way to run xception model on android studio

hello i want to know how i could run this codes to run the application , i want to feed me the steps for that , thank you sir

@dailystudio
Hi, I have trained deep lab on my custom dataset(200*150) with 224 as crop size and during the test, it detects for crop with crop size 224 .
now what I need is to integrate my model on ios application, i was able to successfully convert the model to tflite .but i does not detecte anything i don't get it whats the problem
because when i tried to convert a deeplab pretrained mobilenet mode [llink](http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_train_aug_2018_01_29.tar.gz it works for me on mobile and for my model no ,however i have tested my model (.pb model)with python code and it detects
did you have any idea ??

Hello DailyStudio! I just want to know about official tensorflow's image segmentation example. They are returning result and masked image in their app but how we can get cropped image by using their masked image coordinates?

Deeplab. Occurs on all versions of tensorflow.

Caused by: java.lang.UnsatisfiedLinkError: No implementation found for long org.tensorflow.lite.NativeInterpreterWrapper.createErrorReporter(int) (tried Java_org_tensorflow_lite_NativeInterpreterWrapper_createErrorReporter and Java_org_tensorflow_lite_NativeInterpreterWrapper_createErrorReporter__I) at org.tensorflow.lite.NativeInterpreterWrapper.createErrorReporter(Native Method) at org.tensorflow.lite.NativeInterpreterWrapper.<init>(NativeInterpreterWrapper.java:58) at org.tensorflow.lite.Interpreter.<init>(Interpreter.java:224) at com.dailystudio.deeplab.ml.DeepLabLite.segment(DeepLabLite.java:146) at com.dailystudio.deeplab.SegmentBitmapsLoader.loadInBackground(SegmentBitmapsLoader.java:96) at com.dailystudio.deeplab.SegmentBitmapsLoader.loadInBackground(SegmentBitmapsLoader.java:29) at android.support.v4.content.AsyncTaskLoader.onLoadInBackground(AsyncTaskLoader.java:306) at android.support.v4.content.AsyncTaskLoader$LoadTask.doInBackground(AsyncTaskLoader.java:59) at android.support.v4.content.AsyncTaskLoader$LoadTask.doInBackground(AsyncTaskLoader.java:47) at android.support.v4.content.ModernAsyncTask$2.call(ModernAsyncTask.java:138) at java.util.concurrent.FutureTask.run(FutureTask.java:237) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)  at java.lang.Thread.run(Thread.java:818) 

@dailystudio In the demo you show how the masked image is created. How do I access this image? where is it stored?

My target is to take this masked image and save it.

Do you know how to produce a TFLite file of any arbitary dimension from the deeplab models here:

https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md

I got pretty close.

I have some test code

import numpy as np
import tensorflow as tf

# Load TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="deeplabv3_257_mv_gpu.tflite")
interpreter.allocate_tensors()

# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# Test model on random input data.
input_shape = input_details[0]['shape']
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
interpreter.set_tensor(input_details[0]['index'], input_data)

interpreter.invoke()
output_data = interpreter.get_tensor(output_details[0]['index'])
print(output_data)

Which executes flawlessly

but for my own model that I have converted

(tensorflow-v1.13.1) [samh@apollo-centos6 tmp]$ more speedy.py 
import numpy as np
import tensorflow as tf

# Load TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="speedy.tflite")
interpreter.allocate_tensors()

# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# Test model on random input data.
input_shape = input_details[0]['shape']
input_data = np.array(np.random.random_sample(input_shape), dtype=np.uint8)
interpreter.set_tensor(input_details[0]['index'], input_data)

interpreter.invoke()
output_data = interpreter.get_tensor(output_details[0]['index'])
print(output_data)

(tensorflow-v1.13.1) [samh@apollo-centos6 tmp]$ python speedy.py 
Traceback (most recent call last):
  File "speedy.py", line 17, in <module>
    interpreter.invoke()
  File "/home/samh/anaconda3/envs/tensorflow-v1.13.1/lib/python3.6/site-packages/tensorflow/lite/python/interpreter.py", line 277, in invoke
    self._interpreter.Invoke()
  File "/home/samh/anaconda3/envs/tensorflow-v1.13.1/lib/python3.6/site-packages/tensorflow/lite/python/interpreter_wrapper/tensorflow_wrap_interpreter_wrapper.py", line 109, in Invoke
    return _tensorflow_wrap_interpreter_wrapper.InterpreterWrapper_Invoke(self)
RuntimeError: tensorflow/lite/kernels/depthwise_conv.cc:99 params->depth_multiplier * SizeOfDimension(input, 3) != SizeOfDimension(filter, 3) (0 != 64)Node number 33 (DEPTHWISE_CONV_2D) failed to prepare.

It is erroring on the line .invoke()

the .pb file was created using the export_model.py script here:

https://github.com/tensorflow/models/blob/master/research/deeplab/export_model.py

Using the docs here
https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/export_model.md

It is an xception_65 model

I quantised as follows

tflite_convert --output_file=speedy.tflite --graph_def_file=frozen_graph.pb --inference_type=FLOAT --inference_input_type=QUANTIZED_UINT8 --input_arrays=ImageTensor --input_shapes=1,2049,2049,3 --output_arrays='SemanticPredictions' --std_dev_values=128 --mean_values=127

Which ends cleanly.

Now I know that this will take a while to run on a mobile phone but the end game is to run it on a GPU in OpenGL ES on Linux and Metal on Apple desktop.

Do you have any hints

To repeat here is the error message

  File "/home/samh/anaconda3/envs/tensorflow-v1.13.1/lib/python3.6/site-packages/tensorflow/lite/python/interpreter_wrapper/tensorflow_wrap_interpreter_wrapper.py", line 109, in Invoke
    return _tensorflow_wrap_interpreter_wrapper.InterpreterWrapper_Invoke(self)
RuntimeError: tensorflow/lite/kernels/depthwise_conv.cc:99 params->depth_multiplier * SizeOfDimension(input, 3) != SizeOfDimension(filter, 3) (0 != 64)Node number 33 (DEPTHWISE_CONV_2D) failed to prepare.

Are there plans of introducing application with tflite support for Deeplab. If not , needed help with tflite conversion of recent available pretrained DeeplabV3 MobilenetV2 model.

We were unable to obtain the tflite benchmark of 137 ms on Adreno 530 GPU (One Plus 3 phone ), using the same model in the app( also using our own modified version of official tensorflow gpu classification app ).It comes around 450-500 ms on this device.How did you measure the timing? Is it the time required for 'tflite.run' function, as shown in the app?Or does it have something to do with the phone hardware itself (although specs seem to be same)?

Also, is there any other way to improve the performance, say if we want to directly segment camera frames instead of segmenting local images?The tensorflow site suggests us to use opengl SSBO direclty to avoid io buffers related to CPU-GPU copying.Another tip was to use RGBA (4 channel) frames from camera as input to model.Any idea about how to implement these techniques?

Hi, does the deeplab project support segmentation on live camera video stream? The tensorflow lite project for object detection (https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection) works on camera video and produces a rectangle around the recognized object. I am wondering if the deeplab project can work on camera video (similar to the object detection) and refine the segmentation to show the contour of the object instead of the rectangle?
thanks

Hey,

Loved your library!!

I am looking for a deep learning library/sdk which can be used to remove the background from any image automatically (with quality as good as www.remove.bg).

I tried some image segmentation SDKs with pre-trained models such as Tensorflow Lite & Fritz AI, but the accuracy of the cutout mask was very low, amongst other issues.

Criteria :-

1. Background Removal rather than just Human/Portrait Segmentation

If the foreground consists of person holding a balloon, sittting on a chair, with a pet on his side, then I want all of this to get extracted. Not just the human cutout. The segmentation SDKs I tried are only extracting humans (the chair gets vanished), that too with a very low quality mask (hair gets cut, parts of ear gets cut, etc).

2. Mask quality should be Super-Accurate

I want even the finer details like the hair, delicate clothes, etc to be extracted perfectly.

3. Fast & Lightweight (for mobile phone)

I want to use this technology on mobile phones (in an Android app) which should ideally work even in an offline environment. If this option is difficult to achieve, then plan B would be install the technoloy on our server.

4. Technology
   What technology should I be exploring to achieve this? Is it called image segmentation or the better term would be image matting? (e.g. http://alphamatting.com/eval_25.php)

I have been reading a lot and I am currently lost in the sea of various technologies out there (OpenCV, Deep Matting, Mask RCNN, Instance Segmentation, Detectron2, Tensorflow, Pytorch, etc). I wonder what magic is happening behind the curtains of www.remove.bg

Would your library help me me to achieve what I am looking for? Any help you could provide would be awesome.

Thanks a ton!

TensorFlow Mobile not working, lite has not accurate results

I generated frozen model as per instructions. Now I am wondering how to install this application on android.

Hello @dailystudio . Is there any way of implementing GPU Delegate TFLite with SSBO?? Could you share some related code if you have any idea with the same

Hello!
First of all thanks for this great repo.
Actually I am working on running model on android. I am using Imagenet trained model

https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet
But when I run export_model as you have written It gives me following error.

INFO:tensorflow:Restoring parameters from model/mobilenet_v2_1.0_96.ckpt
2019-06-26 01:35:30.648418: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key logits/semantic/weights not found in checkpoint
2019-06-26 01:35:30.649013: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key logits/semantic/biases not found in checkpoint
2019-06-26 01:35:30.651016: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key image_pooling/weights not found in checkpoint
2019-06-26 01:35:30.652490: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key image_pooling/BatchNorm/moving_variance not found in checkpoint
2019-06-26 01:35:30.654040: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key image_pooling/BatchNorm/moving_mean not found in checkpoint
2019-06-26 01:35:30.655293: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key image_pooling/BatchNorm/gamma not found in checkpoint
2019-06-26 01:35:30.656465: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key image_pooling/BatchNorm/beta not found in checkpoint
2019-06-26 01:35:30.658158: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key concat_projection/weights not found in checkpoint
2019-06-26 01:35:30.660121: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key concat_projection/BatchNorm/moving_variance not found in checkpoint
2019-06-26 01:35:30.662927: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key concat_projection/BatchNorm/moving_mean not found in checkpoint
2019-06-26 01:35:30.665587: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key concat_projection/BatchNorm/gamma not found in checkpoint
2019-06-26 01:35:30.667224: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key concat_projection/BatchNorm/beta not found in checkpoint
2019-06-26 01:35:30.668163: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key aspp0/weights not found in checkpoint
2019-06-26 01:35:30.669490: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key aspp0/BatchNorm/moving_variance not found in checkpoint
2019-06-26 01:35:30.670347: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key aspp0/BatchNorm/moving_mean not found in checkpoint
2019-06-26 01:35:30.671525: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key aspp0/BatchNorm/gamma not found in checkpoint
2019-06-26 01:35:30.672480: W tensorflow/core/framework/op_kernel.cc:1198] Not found: Key aspp0/BatchNorm/beta not found in checkpoint
Traceback (most recent call last):
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_call
return fn(*args)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1329, in _run_fn
status, run_metadata)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key logits/semantic/weights not found in checkpoint
[[Node: save/RestoreV2_271 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_271/tensor_names, save/RestoreV2_271/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "export_model.py", line 186, in
tf.app.run()
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 124, in run
_sys.exit(main(argv))
File "export_model.py", line 177, in main
initializer_nodes=None)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py", line 104, in freeze_graph_with_def_protos
saver.restore(sess, input_checkpoint)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1686, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
feed_dict_tensor, options, run_metadata)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1344, in _do_run
options, run_metadata)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1363, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key logits/semantic/weights not found in checkpoint
[[Node: save/RestoreV2_271 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_271/tensor_names, save/RestoreV2_271/shape_and_slices)]]

Caused by op 'save/RestoreV2_271', defined at:
File "export_model.py", line 186, in
tf.app.run()
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 124, in run
_sys.exit(main(argv))
File "export_model.py", line 163, in main
saver = tf.train.Saver(tf.all_variables())
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1239, in init
self.build()
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1248, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1284, in _build
build_save=build_save, build_restore=build_restore)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 765, in _build_internal
restore_sequentially, reshape)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 268, in restore_op
[spec.tensor.dtype])[0])
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1031, in restore_v2
shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
op_def=op_def)
File "/home/abdullah/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1625, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

NotFoundError (see above for traceback): Key logits/semantic/weights not found in checkpoint
[[Node: save/RestoreV2_271 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_271/tensor_names, save/RestoreV2_271/shape_and_slices)]]

Kindly help me

那个模型是官网发布的，但是怎么自己训练的模型转呢？您那边是否有相关的建议？谢谢！

@dailystudio How did you specify these details in DeepLabLite.java file.

private final static String MODEL_PATH = "deeplabv3_257_mv_gpu.tflite";
    private final static boolean USE_GPU = false;

    private static final float IMAGE_MEAN = 128.0f;
    private static final float IMAGE_STD = 128.0f;
    private final static int INPUT_SIZE = 257;
    private final static int NUM_CLASSES = 2;
    private final static int COLOR_CHANNELS = 3;
    private final static int BYTES_PER_POINT = 4;

I have custom trained TFLite file for semantic segmentation here
resized.tflite.zip

This is how my graph looks:

How do I demo my model using your codebase ?

Hi,
Thanks for this application.
I want to save results, i.e., mask image and cropped image as png, tiff or jpg
into gallery folder again.

What codes should be added into which file?

Crashes with the frozen graph mobilenetv2_coco_voc_trainaug from Deeplab github (https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md):

FATAL EXCEPTION: ModernAsyncTask #3
Process: com.dailystudio.deeplab, PID: 4546
java.lang.RuntimeException: An error occurred while executing doInBackground()
at android.support.v4.content.ModernAsyncTask$3.done(ModernAsyncTask.java:161)
at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:383)
at java.util.concurrent.FutureTask.setException(FutureTask.java:252)
at java.util.concurrent.FutureTask.run(FutureTask.java:271)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
at java.lang.Thread.run(Thread.java:764)
Caused by: java.lang.IllegalArgumentException: No OpKernel was registered to support Op 'Slice' with these attrs. Registered devices: [CPU], Registered kernels:
device='CPU'; T in [DT_BOOL]
device='CPU'; T in [DT_FLOAT]
device='CPU'; T in [DT_INT32]

	 [[Node: SemanticPredictions = Slice[Index=DT_INT32, T=DT_INT64](ArgMax, SemanticPredictions/begin, SemanticPredictions/size)]]
    at org.tensorflow.Session.run(Native Method)
    at org.tensorflow.Session.access$100(Session.java:48)
    at org.tensorflow.Session$Runner.runHelper(Session.java:298)
    at org.tensorflow.Session$Runner.runAndFetchMetadata(Session.java:260)
    at org.tensorflow.contrib.android.TensorFlowInferenceInterface.run(TensorFlowInferenceInterface.java:220)
    at org.tensorflow.contrib.android.TensorFlowInferenceInterface.run(TensorFlowInferenceInterface.java:197)
    at com.dailystudio.deeplab.ml.DeeplabModel.segment(DeeplabModel.java:117)
    at com.dailystudio.deeplab.SegmentBitmapsLoader.loadInBackground(SegmentBitmapsLoader.java:92)
    at com.dailystudio.deeplab.SegmentBitmapsLoader.loadInBackground(SegmentBitmapsLoader.java:27)
    at android.support.v4.content.AsyncTaskLoader.onLoadInBackground(AsyncTaskLoader.java:306)
    at android.support.v4.content.AsyncTaskLoader$LoadTask.doInBackground(AsyncTaskLoader.java:59)
    at android.support.v4.content.AsyncTaskLoader$LoadTask.doInBackground(AsyncTaskLoader.java:47)
    at android.support.v4.content.ModernAsyncTask$2.call(ModernAsyncTask.java:138)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    	... 3 more

Hi, i build this project in Android Studio, and it gets some wrong:
Could not get unknown property 'RELEASE_STORE_FILE' for SigningConfig_Decorated{name=release, storeFile=null, storePassword=null, keyAlias=null, keyPassword=null, storeType=null, v1SigningEnabled=true, v2SigningEnabled=true} of type com.android.build.gradle.internal.dsl.SigningConfig.
thanks !