raluca-scona / staticfusion

StaticFusion

StaticFusion

This repository contains StaticFusion, a method for dense RGB-D SLAM in dynamic environments based on a strategy of simultaneous odometry and dynamic object segmentation.

It has been tested on Ubuntu 16.04.

Publication

If you use StaticFusion in your own work, please cite our paper:

• StaticFusion: Background Reconstruction for Dense RGB-D SLAM in Dynamic Environments. Raluca Scona, Mariano Jaimez, Yvan R. Petillot, Maurice Fallon, Daniel Cremers. IEEE International Conference on Robotics and Automation (ICRA) 2018

Configuration and Dependencies

StaticFusion is organised as a cmake project. The dependencies are:

• OpenCV
• MRPT
• OpenNI2
• Pangolin

Install Steps

1. Ubuntu 16.04

Given a clean installation of Ubuntu, the necessary steps for installing StaticFusion are:

0. Our approach requires a GPU to run, so make sure to install the necessary driver for your graphics card. We only tested the code on Nvidia GPUs so far.

1. Install the necessary libraries (and any required dependencies):

sudo apt-get install cmake libmrpt-dev freeglut3-dev libglew-dev libopencv-dev libopenni2-dev git

2. Download and compile Pangolin.

3. Compile the project, eg:

cd static-fusion 
mkdir build
cd build
cmake ..
make

2. Windows (using CMake and Visual Studio)

0. For the required dependencies: download pre-compiled binaries or the source code (and build it). Don't forget to add the dirs with the binaries to the PATH of Windows.

1. Please build Pangolin from source and disable the flag "MSVC_USE_STATIC_CRT" on CMake. This is required because Pangolin uses static linking by default on Windows and we use dynamic linking.

2. Generate the solution with CMake and compile / run the project you want to try.

Troubleshooting:

• The pangolin window looks completely white -> Resize it once to force it to show the content.
• Runtime error "OpenGL Error: XX (1282) In: ...\source\include\pangolin/gl/gl.hpp". We could solve it following stevenlovegrove/Pangolin#149. Aparently some computers will decide to use the integrated GPU when they actually have a better one even if the integrated GPU does not support some of the functionalities used. If you only have an integrated GPU, I guess it could also happen that it does not support some of the operations implemented in our code...

Running Experiments

There are three executables you can run:

1) StaticFusion-Camera: running off of live RGB-D camera feed, assuming an OpenNI compatible camera.

2) StaticFusion-Datasets: running using the TUM/Freiburg RGB-D datasets in Rawlog format.

• run: ./StaticFusion-Datasets ~/Downloads/rawlog_rgbd_dataset_freiburg1_360/rgbd_dataset_freiburg1_360.rawlog
• You can download these sequeces here. If you would like to perform quantitative evaluation, set bool save_results = true within the StaticFusion-datasets.cpp file to save the estimated trajectory to file. This trajectory can then be evaluated here: http://vision.in.tum.de/data/datasets/rgbd-dataset/online_evaluation.

2) StaticFusion-ImageSequenceAssoc: running using images stored on disk.

• run: ./StaticFusion-ImageSeqAssoc ~/Downloads/ball
• The expected format for images in this case is the same as listed here if one was to use associate.py script to associate color and depth images.

The organisational structure of the dataset should be:

/dataset/rgb/   - folder containing all color images
/dataset/depth/ - folder containing all depth images
/dataset/rgbd_assoc.txt

Where rgbd_assoc.txt contain a list of items of the form:

timestamp1 /rgb/timestamp1.png timestamp2 /depth/timestamp2.png

Should you wish to modify this, the code can be found in the file FrontEnd.cpp, in the methods loadAssoc, loadImageFromSequenceAssoc.

The expected format of the images:

• color images - 8 bit in PNG. Resolution: VGA
• depth images - 16 bit monochrome in PNG, scaled by 1000. Resolution: VGA

Example sequences

ball

selfie

All parameters are optimised for QVGA resolution.

ElasticFusion

We credit ElasticFusion as a significant basis for our work.

staticfusion's Issues

Index out of bounds

Thank you for your excellent work, I have a problem when I run with the rawlog_rgbd dataset,
terminate called after throwing an instance of 'std::logic_error'
what():

=============== MRPT EXCEPTION =============
mrpt::obs::CObservationPtr mrpt::obs::CRawlog::getAsObservation(std::size_t) const, line 157:
Index out of bounds
mrpt::obs::CObservationPtr mrpt::obs::CRawlog::getAsObservation(std::size_t) const, line 164:

Where can I find trajectory result when using-ImageSeqAssoc

Thank you for your efforts making this code.

I have a question.
I ran the code and the code make GUI. However, I cannot find the trajectory result such as text file.
Where can I found it??

StaticFusion with Kinect2

I am trying to run static fusion with the kinect2. When I run the code I get the terminal output "The RGB and the depth frames don't have the same size" and "Eigensolver couldn't find a solution. Pose is not updated" due to the difference in depth frame and rbg frame sizes on Kinect2. Any idea how I can edit the code to fix this?

segmentation fault (core dumped)

I have successfuly built this project. But I found a problem. When I run StaticFusion-Datasets, it may sometimes run successfully and may report an error. The error message is as follows.

segmentation fault (core dumped)

Can you help me fit this problem? Thank you very much.

How do i use it？

I had successfully compiled this program and got the dataset from the page provided by the author. But i didn't know how to use them.

I had used ElasticFusion, as shown below

./ElasticFusion -l dyson_lab.klg

Could you give me some way to use StaticFusion?

Looking forward to your help， thanks！

Cmake Error

Thank you very much for sharing! But I have the following problems when compiling. How can I solve them?

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
pangolin_intf_libs
linked by target "sf_lib" in directory /home/ls-wyf/staticfusion

-- Configuring incomplete, errors occurred!
See also "/home/ls-wyf/staticfusion/build/CMakeFiles/CMakeOutput.log".

About fps

Maybe the difference between CPU...I only get 50ms/frame on my computer. I intend to use cuda to accelerate it and use it for some further usage. Can you give me some suggestions on where to modify? I modify the kmeans algorithm to shorten it into 2ms(before it is 4ms) already.

which versions of opencv and mrpt libraries are needed?

Hi, I tried to compile the code against opencv 3.4 and mrpt 1.9.9. However, this didn't work since there were some missing linking folders (/usr/include/opencv) and libs (mrpt-base). Thus, I ended up using opencv_2.4.9 and mrpt_1.5.

Now I get the following error in the compilation:

[ 81%] Linking CXX executable StaticFusion-ImageSeqAssoc
libsf_lib.a(FrontEnd.o): In function `StaticFusion::loadImageFromSequenceAssoc(std::string const&, std::string const&, unsigned int)':
FrontEnd.cpp:(.text+0x8711): undefined reference to `cv::imread(std::string const&, int)'
FrontEnd.cpp:(.text+0x88a8): undefined reference to `cv::imread(std::string const&, int)'
collect2: error: ld returned 1 exit status
CMakeFiles/StaticFusion-ImageSeqAssoc.dir/build.make:130: recipe for target 'StaticFusion-ImageSeqAssoc' failed
make[2]: *** [StaticFusion-ImageSeqAssoc] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/StaticFusion-ImageSeqAssoc.dir/all' failed
make[1]: *** [CMakeFiles/StaticFusion-ImageSeqAssoc.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

which my guess is that is due to wrong version of the opencv lib. Thus, can you tell us the versions of opecv and mrpt libs that you compiled initially the source code.

Thanks.

glDrawFrustrum typo

In Pangolin a typo has been fixed which makes your program not compile any more if one uses the latest version. In Utils/GUI.h the function pangolin::glDrawFrustrum is called which is now named pangolin::glDrawFrustum

see stevenlovegrove/Pangolin#347

How come c^2/2 is ignored?

Hi

I see that the term c^2/2 of the Cauchy robust penalty is ignored (https://github.com/raluca-scona/staticfusion/blob/master/SegmentationBackground.cpp#L150), but is there any reason why?

Thanks