osrf / tensorflow_object_detector

Tensorflow Object Detector

Tensorflow Object Detector with ROS

Requirements:

Tensorflow and ROS

This guide targets Ubuntu 16.04 and ROS Kinetic

Steps:

To run Default SSD (Single Shot Detection) algorithm:

1. Install ROS: http://wiki.ros.org/kinetic/Installation/Ubuntu

2. Install camera dependencies

   sudo apt-get install ros-kinetic-usb_cam ros-kinetic-openni2-launch

3. Install tensorflow into python virtualenv: https://www.tensorflow.org/install/install_linux

   sudo apt-get install python-pip python-dev python-virtualenv

   virtualenv --system-site-packages ~/tensorflow

   source ~/tensorflow/bin/activate

   easy_install -U pip

   pip install --upgrade tensorflow

4. mkdir ~/catkin_ws/ && mkdir ~/catkin_ws/src/

5. Clone standard Vision messages repository and this repository into catkin_ws/src:

   cd ~/catkin_ws/src

   git clone https://github.com/Kukanani/vision_msgs.git

   git clone https://github.com/osrf/tensorflow_object_detector.git

6. Build tensorflow_object_detector and Vision message

   cd ~/catkin_ws && catkin_make

7. Source catkin workspace's setup.bash:

   source ~/catkin_ws/devel/setup.bash

8. Plug in camera and launch Single Shot Detector (varies per camera, NOTE: object_detect.launch also launches the openni2.launch file for the camera. If you are using any other camera, please change the camera topic in the launch file before launching the file)

   roslaunch tensorflow_object_detector object_detect.launch

   OR

   roslaunch tensorflow_object_detector usb_cam_detector.launch

If you want to try any other ML model:

1. Download any Object Detection Models from the Tensorflow Object detection API and place it in data/models/. You can find the models in tensorflow Object Detection Model Zoo . Extract the tar.gz file.

2. Edit the MODEL_NAME and LABEL_NAME in detect_ros.py. By default it is ssd_mobilenet_v1_coco_11_06_2017 with mscoco_label_map.pbtxt respectively.

tensorflow_object_detector's Issues

non-resource variables are not supported in the long term

Hello, everybody.
I just used this package and at the first time the detect_ros.py node worked. Previously I installed all the python-required libraries without using a python virtual environment and also I used this link trick solution cause my tensorflow version is 2.1.0.

But then, few days later, when I needed to use the package again, I came up with this information when running the node again:

2021-05-12 21:40:06.314810: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alex/imt342_ws/devel/lib:/opt/ros/melodic/lib
2021-05-12 21:40:06.315016: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alex/imt342_ws/devel/lib:/opt/ros/melodic/lib
2021-05-12 21:40:06.315084: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
/usr/lib/python2.7/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/compat/v2_compat.py:88: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
2021-05-12 21:40:10.719210: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2494255000 Hz
2021-05-12 21:40:10.719608: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5570fb475f30 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-05-12 21:40:10.719651: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-05-12 21:40:10.724545: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-05-12 21:40:10.784628: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-12 21:40:10.785798: I tensorflow/compiler/xla/service/platform_util.cc:205] StreamExecutor cuda device (0) is of insufficient compute capability: 3.5 required, device is 3.0
2021-05-12 21:40:10.785924: I tensorflow/compiler/jit/xla_gpu_device.cc:136] Ignoring visible XLA_GPU_JIT device. Device number is 0, reason: Internal: no supported devices found for platform CUDA
2021-05-12 21:40:10.786280: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-12 21:40:10.787618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GT 650M computeCapability: 3.0
coreClock: 0.835GHz coreCount: 2 deviceMemorySize: 1.95GiB deviceMemoryBandwidth: 59.60GiB/s
2021-05-12 21:40:10.787903: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alex/imt342_ws/devel/lib:/opt/ros/melodic/lib
2021-05-12 21:40:10.788094: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alex/imt342_ws/devel/lib:/opt/ros/melodic/lib
2021-05-12 21:40:10.788277: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alex/imt342_ws/devel/lib:/opt/ros/melodic/lib
2021-05-12 21:40:10.788436: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alex/imt342_ws/devel/lib:/opt/ros/melodic/lib
2021-05-12 21:40:10.788592: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alex/imt342_ws/devel/lib:/opt/ros/melodic/lib
2021-05-12 21:40:10.788767: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alex/imt342_ws/devel/lib:/opt/ros/melodic/lib
2021-05-12 21:40:10.788932: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alex/imt342_ws/devel/lib:/opt/ros/melodic/lib
2021-05-12 21:40:10.788956: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1592] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-05-12 21:40:10.788988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-05-12 21:40:10.789007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0 
2021-05-12 21:40:10.789024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N 

And now a I can't have the node working correctly. Even with using a python virtual environment got the same issue.

I'm using:

• ubuntu 18.04
• ros-melodic
• tensorflow version 2.1.0
• python 2.7 (ros-melodic default)

My hardware information is:

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Model name: Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz
Stepping: 9
CPU MHz: 1803.938
CPU max MHz: 2500,0000
CPU min MHz: 1200,0000
BogoMIPS: 4988.51
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 3072K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d

I came up with many resources, on internet, for solving this problem, but I found nothing that solves it.
Can anyone knows how to fix this, please?

tensorflow2 running in ubuntu20.04 ros noetic

will you plan to support tensorflow2 running in ubuntu20.04 ros noetic ?

problem tensorflow ROS node with cam kinect

Hi

I installed tensorflow_object_detector on my system, I use the freenect driver, cam kinect xbox360.
I modified the object_detect.launch file replacing it:
launch/openni2.launch
with
launch/freenect.launch
but I get this warning (RVIZ photo attached) and I don't get photos from the camera. Could you help me?
I have no other errors in the console!

I installed
Ubuntu Mate 16.04, ROS Kinetic, RVIZ, Tensorflow 1.4.1, virtualenv with python 2.7, CV2

Best regards
Giuseppe

use my sensor

Hello, I want to run this project in Melodic according to my sensors.
I have monocular camera and Mainland ARS-408, and also have two sensor ROS driver packages.
Does the fusion part fit?
Looking forward to your reply.
Thank you !

raise RuntimeError('Attempted to use a closed Session.') RuntimeError: Attempted to use a closed Session.

Hey guys,

Thanks for putting this together - it's rare to find ML & ROS tutorials. This is exactly what I needed 👍
I am trying to run the usb_cam example: roslaunch tensorflow_object_detector usb_cam_detector.launch and it looks like the program runs fine until this line: (boxes, scores, classes, num_detections) = sess.run([boxes, scores, classes, num_detections], feed_dict={image_tensor: image_np_expanded})

The error below is what I get. It loops over in image_cb and prints the following error every time it gets to the line mentioned above:

[ERROR] [1539275044.021065]: bad callback: <bound method detector.image_cb of <__main__.detector instance at 0x7fc7073b5488>>
Traceback (most recent call last):
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback
    cb(msg)
  File "/home/gal/Tensorflow_ws/src/tensorflow_object_detector/scripts/detect_ros.py", line 121, in image_cb
    feed_dict={image_tensor: image_np_expanded})
  File "/home/gal/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 887, in run
    run_metadata_ptr)
  File "/home/gal/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1033, in _run
    raise RuntimeError('Attempted to use a closed Session.')
RuntimeError: Attempted to use a closed Session.

Wrong Centroid calculus

Hello, thanks for publishing this node.
I noticed that the centroid of the detections are not correctly calculated. From the detect_ros.py#L143 it details

obj.bbox.center.x = int((dimensions[1] + dimensions [3])*image_height/2)
obj.bbox.center.y = int((dimensions[0] + dimensions[2])*image_width/2)

To "debug" it I manually added the following at the end of the "object_predict" method

cv2.circle (image_np,(obj.bbox.center.x,obj.bbox.center.y),5,(0,0,255), 5)

image_width and image_height are not correctly used, they should be switched. I mean

obj.bbox.center.x = int((dimensions[1] + dimensions[3])*image_width/2)
obj.bbox.center.y = int((dimensions[0] + dimensions[2])*image_height/2)

regards,

Issue with intel Real sence Camera

I am using intel Renascence camera. I changed the video device value to /dev/video13 in usb_cam_detector launch file as I can access rgb video stream through this value. But I am getting

[detect_ros-1] process has died [pid 11767, exit code 1, cmd /home/smart_drone/catkin_ws/src/tensorflow_object_detector/scripts/detect_ros.py image:=/usb_cam_node/image_raw __name:=detect_ros __log:=/home/smart_drone/.ros/log/5b394e80-18a8-11e9-9950-32da8691a29a/detect_ros-1.log].

Error parsing message

I have run my own model but there are some wrong as this:
od_graph_def.ParseFromString(serialized_graph)
google.protobuf.message.DecodeError: Error parsing message

OS: Ubuntu 20.04
ROS: noetic
TF: 2.4.1
Python: 3.8.5

Help me Please.

ubuntu 18.10 and ROS Melodic?

Will this package work with ubuntu 18.10 and ROS Melodic?

Instructions for updating: non-resource variables are not supported in the long term