natalnet-lpr / rgbd_rtk

RGB-D Reconstruction Toolkit

Natalnet/LPR RGB-D Reconstruction Toolkit

Introduction

A set of utility code built by the Natalnet/LPR group for 3D reconstruction applications using RGB-D (e.g. Microsoft Kinect) cameras.

Requirements

• OpenCV: www.opencv.org (version = 3.X)
• OpenCV-contrib: https://github.com/opencv/opencv_contrib (Same version as OpenCv)
• PCL: www.pointclouds.org (version 1.8)
• G2O: https://github.com/RainerKuemmerle/g2o/releases (version 20200410)
• ARUCO: https://sourceforge.net/projects/aruco/files/ (Version = 3.X)
• Eigen: http://eigen.tuxfamily.org/index.php?title=Main_Page (Version = 3.X)

It is recommended to install both dependencies from source. The following two links will redirect you to their official tutorials to do so.

Install OpenCV on Linux

Compiling PCL from source on Linux

Building On Ubuntu 20.04

First, get the newest version of this repository by using the follwing command on your working directory.

git clone https://github.com/natalnet-lpr/rgbd_rtk.git

Go to the project folder

cd rgb_rtk

The fastest way to build the project and it requirements is using our setup script.

bash setup.sh -a -v

This gonna take a while (some hours depending on your computer settings).

After this, you must be able to execute the program files at the build/bin folder

Others Distros

First, get the newest version of this repository by using the follwing command on your working directory.

cd /my_working _directory 

git clone https://github.com/natalnet-lpr/rgbd_rtk.git

Create a directory called build, and change directory to it.

mkdir build

cd build/

Inside of the build directory, run the cmake command so that it will create the necessary makefiles.

cmake ..

Then, compile the rgbt_rtk by using the make command. The -j4 is optional but it is recommended. With it your code will be compiled in 4 parallel tasks.

make -j4

sudo make install

License

This code is distributed under the terms of the BSD License.

Authors

Natalnet Laboratory for Perceptual Robotics, Federal University of Rio Grande do Norte, Brazil.

Contact: bruno.silva AT ect.ufrn.br

rgbd_rtk's Issues

Functions param documentation are wrong

Measure tracking quality with precision/recall

Update the code that generates the precision/recall curves from thesis to evaluate other feature keypoints/descriptors.

Memory Leaks error in Visualizer

To reproduce:
Just need to have a reference to "ReconstructionVisualizer"
ReconstructionVisualizer visualizer; For example.

• It is not a big problem since the error only occurs after the program ends.
• There is no need to use any method of ReconstructionVisualizer class to see the error.
• it is likely to be an error with deallocation of memory

Visualization of graph vertices, edges and keyframes

Add functionalities to the visualization module to support visualising graph vertices, edges and keyframes.

Create a check if the node(key) in configfile.yaml is empty

Support KITTI/Stereo datasets

Include the implementation from Ferreira and Vanessa works that enables the RGB-D RTK to process KITTI/Stereo datasets.

Added KeyFrame to visualizer and a function to visualizer a graph

Create FindOpenFABMAP.cmake file

Arren Glover implemented a open source version in C++ of OpenFABMAP, this implementation can be very useful because the source code was wrote using OpenCV, for link his library with our library is necessary to create a .cmake file to found the library.

New flag in marker finder in order to select witch marker pose function use

There is three functions to calculate the marker pose, "setMarkerPosesLocal", "setMarkerPosesGlobal" and "setMarkerPointPosesGlobal". The way it works now it uses all three functions, creating a new parameter in "detectMarker" to choose which function to use is more logical.

Implement visualization module with OpenCV backend

Implement visualization module using OpenCV Viz functions.

Reject wrong matches in motion estimator

Add a method in motion class to reject wrong matches using RANSAC.

The method estimate is currently implemented with RANSAC but is not removing wrong matches.
Therefore, we could store the indices of the inlier/outlier correspondences in a vector and update both vectors by removing positions marked as wrong, after calling the method.

Implement a configuration loader for Visual Odometry/SLAM

The configuration loader should be a new class. Instances of this class should be used as an instance attribute of the Visual Odometry class.

Design keyframe class

A keyframe class is needed to represent the needed information (camera pose and point cloud) for the optimization process. This class should inherit from the G2O vertex class.

Memory leaks provide by FeatureDetector::detectAndCompute()

when use the function FeatureDetector::detectAndCompute(), even when the object is deleted the program have a little memory leak, something around 400 bytes, but is necessary fix it.

Estimate motion with multiple hypothesis

Motion should be estimated from multiple previews frames, yielding thus multiple motion hypothesis.

Update the new dependency in Read.me

We only cite three dependencies in Read.me, Update to show all of them.

Segmentation Fault in Marker Finder

Implement a system logger

A system logger should be visible/usable by all other classes as a singleton.
The logger should be used to log events, warnings, debug messages, errors, etc.

Remove warnings

A lot of minor warnings such as comparison between signed and unsigned int

Eigen compatibilty issues

Projects using Eigen should comply to several prerequisites, listed e.g. in:

• https://github.com/ethz-asl/eigen_catkin/wiki/Eigen-Memory-Issues#memory-misalignment
• http://eigen.tuxfamily.org/dox-devel/group__TopicUnalignedArrayAssert.html

Currently, the library is not working properly on Mac OS High Sierra (with seg. faults and/or wrong resulting matrices).

Therefore, classes using Eigen must be refactored to support these prerequisites.

Implement pose update of reference frames in visualizer class

ICP odometry

1. Check with Luiz Felipe if this is already done
2. Test it, assess its quality
3. Do the same with ICP + Visual Odometry

Organize ROS wrapper code

Organize ROS wrapper code according to functionality:

-Geometry (relative transformations between poses, point transformation, etc.) code should be in module common/geometry
-ARUCO specific code should be in module SLAM (marker detection)
-All ROS variables should be in a ROSWrapper class as instance attributes

Check and fix the use of EventLogger global var. (use of extern, etc.)

Implement a visual memory class with OpenFABMAP

Implement a version of visual memory class using OpenFABMAP.

Keyframe detection algorithm

Implement a keyframe detection algorithm.

Check with Luiz Felipe if this is already done.

Integrate the RGB-D grabber code

Use old implementations to integrate this code into the IO module.

Update aruco to version 3

Tracking by detection algorithm

Implement a tracking by detection algorithm for the tracker interface.
This implementation should be flexible enough to support combinations of feature detector/descriptor extractors (SIFT, SURF, ORB, etc.)

Document the whole library

Document RGB-D RTK using Doxygen.

Implement a Pose Optimizer (SLAM Solver) class

This class should have an interface to insert/remove nodes/vertices from the graph, allowing tackling SLAM with a high abstraction level.

Allow parameter configuration in MotionEstimation

Allow MotionEstimation class to set parameter values through constructor and set methods.

motion_estimator_test.cpp segmentation fault error

Segmentation Fault ocorre nesse bloco de código (linhas 95-100)
if (i > 0)
{
trans = motion_estimator.estimate(tracker.prev_pts_, prev_cloud, tracker.curr_pts_, curr_cloud);
pose = pose * trans;
}

Dataset: ar_mm_lab1
config_files: marker_detection.yaml

Add Keyframe in slam_solver and reconstruction_visualizer

Implementar SLAM SOLVER com multiplos Marcadores

Update old implementation of tracking strategies

This issue possibly involves designing a Abstract Base Class for the KLT Tracker and
implementing concrete versions with KLT, KLT with Tracking Windows, etc.

Add date/time to log messages

Add a prefix to all messages in the format YYYY-MM-DD HH:MM:SS in such a way that the log message
[MotionEstimatorRANSAC] INFO: inlier ratio: 0.727273

would appear
[2020-05-15 15:20:32 MotionEstimatorRANSAC] INFO: inlier ratio: 0.727273

and a message
[2020-05-15 15:20:32 common::EventLogger] Session start

would be printed before
[[2020-05-15 15:20:33 common::EventLogger] INFO: opening file log.txt

Estimate motion between current frame and last keyframe

The same motion_estimator algorithm can be used to estimate motion between two arbitrary frames.
In this enhancement, the motion should be estimated between the current frame and the last extracted keyframe