baidu / ice-ba

ICE-BA

ICE-BA: Incremental, Consistent and Efficient Bundle Adjustment for Visual-Inertial SLAM

We present ICE-BA, an incremental, consistent and efficient bundle adjustment for visual-inertial SLAM, which takes feature tracks, IMU measurements and optionally the loop constraints as input, performs in parallel both local BA over the sliding window and global BA over all keyframes, and outputs camera pose and updated map points for each frame in real-time. The main contributions include:

• a new BA solver that leverages the incremental nature of SLAM measurements to achieve more than 10x efficiency compared to the state-of-the-arts.
• a new relative marginalization algorithm that resolves the conflicts between sliding window marginalization bias and global loop closure constraints.

Beside the backend solver, the library also provides an optic flow based frontend, which can be easily replaced by other more complicated frontends like ORB-SLAM2.

The original implementation of our ICE-BA is at https://github.com/ZJUCVG/EIBA, which only performs global BA and does not support IMU input.

Authors: Haomin Liu, Mingyu Chen, Yingze Bao, Zhihao Wang
Related Publications:
Haomin Liu, Mingyu Chen, Guofeng Zhang, Hujun Bao and Yingze Bao. ICE-BA: Incremental, Consistent and Efficient Bundle Adjustment for Visual-Inertial SLAM. (Accepted by CVPR 2018).PDF.
Haomin Liu, Chen Li, Guojun Chen, Guofeng Zhang, Michael Kaess and Hujun Bao. Robust Keyframe-based Dense SLAM with an RGB-D Camera [J]. arXiv preprint arXiv:1711.05166, 2017. [arXiv report].PDF.

1. License

Licensed under the Apache License, Version 2.0.
Refer to LISENCE for more details.

2. Prerequisites

We have tested the library in Ubuntu 14.04 and Ubuntu 16.04.
The following dependencies are needed:

boost

sudo apt-get install libboost-dev libboost-thread-dev libboost-filesystem-dev

Eigen

sudo apt-get install libeigen3-dev

Glog

https://github.com/google/glog

Gflags

https://github.com/gflags/gflags

OpenCV

We use OpenCV 3.0.0.
https://opencv.org/

Yaml

https://github.com/jbeder/yaml-cpp

brisk

https://github.com/gwli/brisk

3. Build

cd ice-ba
chmod +x build.sh
./build.sh

4. Run

We provide examples to run ice-ba with EuRoC dataset.

run ICE-BA stereo

Run ICE-BA in stereo mode. Please refer to scripts/run_ice_ba_stereo.sh for more details about how to run the example.

run ICE-BA monocular

Run ICE-BA in monocular mode. Please refer to scripts/run_ice_ba_mono.sh for more details about how to run the example.

run back-end only

Front-end results can be saved into files. Back-end only mode loads these files and runs backend only.
Please refer to scripts/run_backend_only.sh for more details about how to run the example.

5. Contribution

You are very welcome to contribute to ICE-BA. Baidu requires the contributors to e-sign CLA (Contributor License Agreement) before making a Pull Request. We have the CLA binding to Github so it will pop up before creating a PR.

ice-ba's Issues

Math bug in EigenMatrix3x3f variant of GetRodriguesJacobianInverse

I believe the following line should be + (1.0f + cosf(th)) ... (plus, not minus).

ICE-BA/Backend/Geometry/Rotation.h

Line 1340 in b004bb5

- (1.0f + cosf(th)) / (2.0f * th * sinf(th))) * e_S * e_S);

Reference: unnamed Equation between 9 and 10 in https://arxiv.org/pdf/1512.02363.pdf

Remove an unnecessary null pointer check

An extra null pointer check is not needed in a function like the destructor for the class “Viewer”.

similarities with a recent bundle adjustement procedure

DI you find similarities between ICE-BA and the bundle adjustment procedure proposed in this monocular slam paper?

numerical stability

hi, i want to ask for the numerical stability in optimization when constructing the sparse structure Sx = d

Evalutation Scripts, RMSE 0% on all sequences, trajectory file consists only of small number of frames

Hi, I have two issues with the code base.

1. I have got to compare accuracy of ICE-BA with some different front-end. But I could not find any evaluation script for computing ATE.
2. I went through the code and found that back-end consists of RMSE calculations. However when I run it, I always get zero LBA and GBA error.

I get the following output:

../bin/back_end ../config/config_of_stereo.txt
Loaded '../config/config_of_stereo.txt'
Found '/home/isl-server/vo_seqs/vo_sequences/MH02/mav0/cam0/data/1403637010601666560.png'
Found '/home/isl-server/vo_seqs/vo_sequences/MH02/dat/1403637010601666560.png.dat'
Found '/home/isl-server/vo_seqs/vo_sequences/MH02/mav0/cam1/data/1403637010601666560.png'
Loaded '/home/isl-server/vo_seqs/vo_sequences/MH02/./calibration.dat'
Tcu = 0.014865 0.999557 -0.025774 0.065223
-0.999881 0.014967 0.003756 -0.020706
0.004140 0.025716 0.999661 -0.008055
Tlr = 0.999997 -0.002317 -0.000343 0.110074
0.002312 0.999898 -0.014091 -0.000157
0.000376 0.014090 0.999901 0.000889
K = 458.653992 457.295990 367.214996 248.375000
d = -0.283408 0.073959 0.000194 0.000018 0.000000 0.000000 0.000000 0.000000
Kr = 457.587006 456.134003 379.998993 255.238007
dr = -0.283684 0.074513 -0.000105 -0.000036 0.000000 0.000000 0.000000 0.000000
Loaded '../config/config_of_stereo.txt'
3040 / 3040 = 100.000000%

RMSE
LBA = 0.000000 / 73.761749 = 0.000000%
GBA = 0.000000 / 73.761749 = 0.000000%
Saved '/home/isl-server/vo_seqs/vo_sequences/MH02/../RMSE_stereo/MH02_1.txt'

Time
LBA = 7206.813965 / 3040 = 2.370662 ms
GBA = 2642.825439 / 315 = 8.389922 ms

Kindly provide solution as soon as possible as we need to meet a near deadline.

Thank you

../bin/ice_ba: symbol lookup error: ../bin/ice_ba: undefined symbol: _ZN6google21kLogSiteUninitializedE

Holle everyone, recently I met a trouble when I tried to run ICE-BA

ginger@ginger-OptiPlex-9020:~/SLAM/ICE-BA/scripts$ ./run_ice_ba_mono.sh mkdir: cannot create directory ‘/home/ginger/Dataset/MH_01_easy/result’: File exists ../bin/ice_ba --imgs_folder /home/ginger/Dataset/MH_01_easy/MH_01_easy --start_idx 0 --end_idx -1 --iba_param_path ../config/config_of_mono.txt --gba_camera_save_path /home/ginger/Dataset/MH_01_easy/result/MH_01_easy.txt ../bin/ice_ba: symbol lookup error: ../bin/ice_ba: undefined symbol: _ZN6google21kLogSiteUninitializedE
so,did someone meet the same error? how to solve it?

Why Embedded Point Iteration dosen't work?

Why EPI is not used by default? When I use EPI in LBA and GBA, it seems that running speed is not improved. Dose anyone has an idea about it? Thanks a lot.

about local BA and global BA

hi, haomin
I'm very happy to read your papers. In RKD-SLAM, u report Efficient Incremental BA, which do local BA and global BA seamlessly without switch。
but In ICE-BA, u split local BA and global BA in two thread. why?

add the essential comments in project

i found the code understanding is very difficult, could you help add some essential comments?

Fixed KeyFrame in Local BA?

I was trying to do double-window optimization with ICE-BA, since it is the most efficient & incremental solver so far. However, according to my quick glance of the code, it doesn't seem to support fixed keyframe in local BA (and map points thereafter, because of the inverse-depth parameterization).

I was wondering if there's any (relatively) easy modification to fix a subset of keyframes in local BA, while the rest of them are flexible. Appreciate any suggestion!

Can it run on an embedded platform？

Can it run on an embedded platform？
thank you

windows version

Will it have a windows version? Or did anyone have try build it on windows?

ISAM2 backend implementation.

In the paper it is mentioned that you compare the ICE-BA system to using the frontend with ISAM2. I am trying to do some of my own comparisons and was wondering if that code would be available for use.

about callback function

In the callback function, slidingwindow.Xs are obtained, I think the Xs represent the word 3D points. is it right?
According to the GetSlidingWindow function , the Xs are obtained by C.ApplyInversely(x). does it mean Rotation_KeyFrame *(x-Translation_KeyFrame)?
but when i want to get image coordinates for the feature point of each frame by Rotation_frame.inverse()*Xs-Translation_frame, I found the points drift when the image motion happen. anyone can help me for this question?

mistake in project?

inline void Project(const Rigid3D &T12, const Point2D &x1, LA::Vector2f &x2,
LA::Vector2f &Jx2) const {
const xp128f t = xp128f::get(x1.x(), x1.y(), 1.0f, u());
const float d12 = 1.0f / (T12.r20_r21_r22_tz() * t).vsum_all();
x2.x() = (T12.r00_r01_r02_tx() * t).vsum_all() * d12;
x2.y() = (T12.r10_r11_r12_ty() * t).vsum_all() * d12;
Jx2.x() = (T12.tx() - x2.x() * T12.tz()) * d12;
Jx2.y() = (T12.ty() - x2.y() * T12.tz()) * d12;
}

i think the jacobian should be

Jx2.x() = (T12.tx() - d12 * x2.x() * T12.tz()) * d12;
Jx2.y() = (T12.ty() - d12 * x2.y() * T12.tz()) * d12;

whether this project support monocular Visual inertial SLAM ?

I noticed that the project has a macro named CFG_STEREO, whether this project support monocular Visual inertial SLAM ?

Convert to asl timestamp error

I had a problem running ice_ba_mono.sh
It shows [E0306 14:40:59.379793 3010 EuRoC_test.cpp:228] Convert to asl timestamp error

what is CVD?

hi, i want to know what is cvd in CMakeLists?

After successful compilation, it cannot run. I use opencv3.3.0

When I use the -- help command, the prompt section is wrong，Is it the opencv version? Thank you very much!

how to export trajectory form tango phab 2 ?

hi @liuhaomin @nwind 膜拜一下大神们，你们好，我想请教一下，你们的tango 轨迹数据是 用什么方式或者app导出来的呢``使用 rtab-map app?

An error when implemented with my own images and imu data in stereo mode

Hi
Recently I have tested ICE-BA with my own data with the same format as EuRoC dataset in stereo mode. I just used the command ./run_ice_ba_stereo.sh to run the script without changing any default parameter except my own file name and path. But here came an error as below after calibration.dat was created successfully.
*** Aborted at 1545561811 (unix time) try "date -d @1545561811" if you are using GNU date ***
PC: @ 0x7fb8c5747fdf IMU::PreIntegrate()
*** SIGSEGV (@0x1c) received by PID 1734 (TID 0x7fb8c5dea980) from PID 28; stack trace: ***
@ 0x7fb8c4d44390 (unknown)
@ 0x7fb8c5747fdf IMU::PreIntegrate()
@ 0x7fb8c5748166 IMU::PreIntegrate()
@ 0x7fb8c56a1ed9 LocalBundleAdjustor::SynchronizeData()
@ 0x7fb8c569d4a0 LocalBundleAdjustor::Run()
@ 0x7fb8c55df707 IBA::Solver::PushCurrentFrame()
@ 0x41523f main
@ 0x7fb8c1560830 __libc_start_main
@ 0x417419 _start
I suppose this Segment Fault is probably related to IMU settings or calibration. Is there anyone who met such kind of error before? I will be of great appreciation if someone could help me solve this problem or analyze how it occurred.

realtime running

how to run this real-time ?

brisk

Hi,
when i compile the codes, i find that it can not find some include files about brisk,
such as #include <brisk/internal/score-calculator.h>,
so how to find them? Thanks.

how to deal with the ill-conditioned Hession matrix

Thank you for sharing! I read your code, and I saw that you use GN or DL to solve the optimization problem. As I know, GN and DL method are not stable, we use LM in a general way. What extra work do you make to solve this problem to use GN or DL happily?

Clarification regarding feature depth updates

Hello,
I'd like to use ICE-BA with a PNP-style monocular frontend, in which I have metric camera poses, and landmarks with known depth.

1. I feed the 1st keyframe to ICE-BA with the corresponding IMU measurements, 3D camera pose prior, 2D features and their 3D priors.The LBA optimized camera pose roughly remains the same (as expected), while the 3D landmarks try to minimize the re-projection error in the given frame, hence their 3D position change significantly. This is clear. I verified that the old re-projection errors are in the order of 10^-2 (on the frontend), while the ICE-BA new re-projection errors drop below 10^-5 (only for the frame inserted).
2. Then I go ahead, and add a 2nd frame (or more), let it be either a keyframe or a regular frame, having overlapping feature measurements with the 1st frame. What I notice is that the camera pose again roughly remains the same as the input (as expected), but the updated landmarks' 3D position result in large re-projection error.

How do you explain this latter behavior?

Is loop closure supported? If so, how to turn loop closure on?

Options to turn off backend

@liuhaomin @wangjio

• I believe the global ba is on by default. Is there an option to run VIO only?

• If I focus on the speed of the VIO (the estimation part, not including image processing), which function should I time to get a good evaluation?

Thank you!

Support of stereo mode without IMU?

Although ICE BA is designed for VIO, is it possible to run without IMU? This has a wider application,like stereo camera rigs and RGBD camera (depth could be converted to stereo mode).

recompile with -fPIC

when i run ./build.sh，there is an error
/usr/bin/ld: /usr/local/lib/libyaml-cpp.a(convert.cpp.o): relocation R_X86_64_32 against `.bss' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/libyaml-cpp.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
Frontend/CMakeFiles/OF.dir/build.make:378: recipe for target 'Frontend/libOF.so' failed
make[2]: *** [Frontend/libOF.so] Error 1
CMakeFiles/Makefile2:99: recipe for target 'Frontend/CMakeFiles/OF.dir/all' failed
make[1]: *** [Frontend/CMakeFiles/OF.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

what is going on?
what can i do?

how to run ice-ba

when i run the program, i meet the follow problem.
/home/robot/myslam/ICE-BA/bin/ice_ba --imgs_folder /media/robot/file/dataset/EuRoC/MH/MH_01_easy --start_idx 0 --end_idx 1000 --iba_param_path /home/robot/myslam/ICE-BA/config/config_of_mono.txt --gba_camera_save_path /media/robot/file/dataset/EuRoC/MH/result/MH_01_easy.txt
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
*** Aborted at 1562862214 (unix time) try "date -d @1562862214" if you are using GNU date ***
PC: @ 0x7fe4a137e428 gsignal
*** SIGABRT (@0x3e800004b9a) received by PID 19354 (TID 0x7fe4a7357780) from PID 19354; stack trace: ***
@ 0x7fe4a137e4b0 (unknown)
@ 0x7fe4a137e428 gsignal
@ 0x7fe4a138002a abort
@ 0x7fe4a1cc184d __gnu_cxx::__verbose_terminate_handler()
@ 0x7fe4a1cbf6b6 (unknown)
@ 0x7fe4a1cbf701 std::terminate()
@ 0x7fe4a1cbf919 __cxa_throw
@ 0x7fe4a1cbfebc operator new()
@ 0x413ebd std::__cxx11::basic_string<>::_M_construct<>()
@ 0x41b96c boost::filesystem::operator<< <>()
@ 0x4101bf main
@ 0x7fe4a1369830 __libc_start_main
@ 0x413cc9 _start
@ 0x0 (unknown)
./run_ice_ba_mono.sh: line 12: 19354 Aborted (core dumped) /home/robot/myslam/ICE-BA/bin/ice_ba --imgs_folder /media/robot/file/dataset/EuRoC/MH/MH_01_easy --start_idx 0 --end_idx 1000 --iba_param_path /home/robot/myslam/ICE-BA/config/config_of_mono.txt --gba_camera_save_path /media/robot/file/dataset/EuRoC/MH/result/MH_01_easy.txt

How to integrate ICE-BA with other frontends

Dear @wangjio @liuhaomin @mingyux,

I am trying to integrate ICE-BA backend with a different Frontend framework (vilib frontend by RPG lab in Zurich).

The vilib framework offers the featured detection and tracking part of a 2D-to-2D VIO approach.

My question is what are the steps to integrate such a frontend with your backend? More precisely, if I have computed the feature detection and feature tracking already, how can I integrate these with you backend.

It seems in the example you furnished, it only shows how you use your frontend in order to detect features and then create the .dat files. But I already have the features.

Any direction to point me in the right way would be much appreciated :)

How to run multi thread ICE-BA (run local/global BA in parallel)

ICE-BA/Backend/IBA/IBA.h

Line 38 in b004bb5

void Create(const Calibration &K, const int serial = IBA_SERIAL_NONE,

the serial parameter controls running mode, specifically:

• IBA_SERIAL_NONE is the default parameter, both LBA and GBA runs in background, calling PushFrame returns directly
• IBA_SERIAL_LBA makes LBA serial, the GBA still runs in background, calling PushFrame will block until LBA finished
• IBA_SERIAL_LBA|IBA_SERIAL_GBA both serial, calling PushFrame will block until both LBA and GBA finished

NOTE In the demo apps, we create solver with IBA_SERIAL_LBA|IBA_SERIAL_GBA.

license for contributions

Can you confirm that contributions to your project are accepted under Apache-2.0?

thank you.

No image files for detection?

I had a problem running the program.
It shows E0727 11:31:06.477556 18713 EuRoC_test.cpp:367] No image files for detection.
But I've put the dataset "MH_01_easy" into the folder "EuRoC". Who can help me solve this problem?

set parameters constant

Hi,
Thanks for your impresive work.
When I run ice-ba for a long time, the GBA nees amount of computer resourse. How can I solve this problame? If I want to set old parameters constant, where shall I change in the code?
Thanks!

Running with different SLAM algorithms

I am trying to pipeline a different slam algorithm into this bundle adjustment implementation. Is there a straightforward method of inputting poses into this and just running bundle adjustment?

When will the code be available

Hi, authors ,

When will the code be available?