rpng / calc

Convolutional Autoencoder for Loop Closure

   ******      **     **         ****** 
  **////**    ****   /**        **////**
 **    //    **//**  /**       **    // 
/**         **  //** /**      /**       
/**        **********/**      /**       
//**    **/**//////**/**      //**    **
 //****** /**     /**/******** //****** 
  //////  //      // ////////   //////

Convolutional Autoencoder for Loop Closure. A fast deep learning architecture for robust SLAM loop closure, or any other place recognition tasks. Download our pre-trained model with ./DeepLCD/get_model.sh, or train your own!

This repo is separated into two modules. TrainAndTest for training and testing models, and DeepLCD, which is a C++ library for online loop closure or image retrieval. See their respective READMEs for details.

If you use this code in any publication, please cite our paper. The pdf can be found here, and the reference should be in this format:

@InProceedings{Merrill2018RSS,
  Title                    = {Lightweight Unsupervised Deep Loop Closure},
  Author                   = {Nathaniel Merrill and Guoquan Huang},
  Booktitle                = {Proc. of Robotics: Science and Systems (RSS)},
  Year                     = {2018},
  Address                  = {Pittsburgh, PA},
  Month                    = jun # { 26-30, }
}

calc's Issues

build error

Hello, thank you for your research. When I build DeepLCD, I get this error:
haosu@haosu-B85-HD3:~/LCD/calc/DeepLCD/build$ make
[ 5%] Performing update step for 'googletest'
当前分支 master 是最新的。
[ 11%] Performing configure step for 'googletest'
-- Configuring done
-- Generating done
-- Build files have been written to: /home/haosu/LCD/calc/DeepLCD/build/src/googletest-build
[ 16%] Performing build step for 'googletest'
[ 25%] Built target gtest
[ 50%] Built target gmock
[ 75%] Built target gmock_main
[100%] Built target gtest_main
[ 22%] No install step for 'googletest'
[ 27%] Completed 'googletest'
[ 44%] Built target googletest
[ 55%] Built target deeplcd
[ 66%] Built target demo
make[2]: *** No rule to make target 'src/googletest-build/googlemock/gtest/libgtest.a', needed by 'deeplcd-test'。 停止。
CMakeFiles/Makefile2:211: recipe for target 'CMakeFiles/deeplcd-test.dir/all' failed
make[1]: *** [CMakeFiles/deeplcd-test.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2
I use cuda 8.0 and https://github.com/BVLC/caffe , can you help me?

are there any papers related to the library?

from pydbow2 import pyDBoW2，: cannot import name pyDBoW2

When I compiled and build DBOW2, I want to do from pydbow2 import pyDBoW2,Console display cannot import name pyDBoW2, how I to dealwith it.

make error

When I make the DeepLCD code, errors occur as the follow:

[ 50%] Building CXX object CMakeFiles/deeplcd.dir/src/deeplcd/deeplcd.cpp.o
In file included from /home/cyc/caffe/build/install/include/caffe/blob.hpp:9:0,
from /home/cyc/caffe/build/install/include/caffe/caffe.hpp:7,
from /home/cyc/calc-master/DeepLCD/include/deeplcd/deeplcd.h:1,
from /home/cyc/calc-master/DeepLCD/src/deeplcd/deeplcd.cpp:1:
/home/cyc/caffe/build/install/include/caffe/proto/caffe.pb.h:12:2: error: #error This file was generated by a newer version of protoc which is
#error This file was generated by a newer version of protoc which is
^
/home/cyc/caffe/build/install/include/caffe/proto/caffe.pb.h:13:2: error: #error incompatible with your Protocol Buffer headers. Please update
#error incompatible with your Protocol Buffer headers. Please update
^
/home/cyc/caffe/build/install/include/caffe/proto/caffe.pb.h:14:2: error: #error your headers.
#error your headers.
^
In file included from /home/cyc/caffe/build/install/include/caffe/blob.hpp:9:0,
from /home/cyc/caffe/build/install/include/caffe/caffe.hpp:7,
from /home/cyc/calc-master/DeepLCD/include/deeplcd/deeplcd.h:1,
from /home/cyc/calc-master/DeepLCD/src/deeplcd/deeplcd.cpp:1:
/home/cyc/caffe/build/install/include/caffe/proto/caffe.pb.h:23:35: fatal error: google/protobuf/arena.h: no such file or directory
compilation terminated.
CMakeFiles/deeplcd.dir/build.make:62: recipe for target 'CMakeFiles/deeplcd.dir/src/deeplcd/deeplcd.cpp.o' failed
make[2]: *** [CMakeFiles/deeplcd.dir/src/deeplcd/deeplcd.cpp.o] Error 1
CMakeFiles/Makefile2:136: recipe for target 'CMakeFiles/deeplcd.dir/all' failed
make[1]: *** [CMakeFiles/deeplcd.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2

how to solve it?

New File

Hello, can you write a code to make it run in gitpod.io @rpng-guest

Cannot use GPU in CPU-only Caffe

Hello, there is an error when I run TrainAndTest, how can I solve it?

WARNING: Logging before InitGoogleLogging() is written to STDERR
F0618 21:05:34.035341 6893 common.cpp:66] Cannot use GPU in CPU-only Caffe: check mode.
*** Check failure stack trace: ***
Aborted (core dumped)

implementation about the landmark-based place recognition

Docker support for managing dependencies

It would really help if you could setup docker support for this project.

• The TrainAndTest python module requires python2 and a host of dependencies. If I could pull a docker container with all these dependencies, it would make life much easier - I could docker pull and run python main.py db ... or python main.py net ... with no additional effort
• Similarly, you could build off of a ROS docker container for the DeepLCD C++ code. Having a docker container would help tremendously.

If you provide a Dockerfile, the typical usage for the user would be:

$ git clone https://github.com/rpng/calc
$ cd calc/TrainAndTest # Assuming there is a Dockerfile here
$ docker build calc/train:1.0 .
$ docker run -it --volume="/home/foo/bar/calc:/home/root/calc" calc/train:1.0 bin/bash
root@ubuntu:/# cd /home/root/calc/TrainAndTest
root@ubuntu:/# python main.py -h

Here is an example docker file for voxblox.

deep model in SLAM

Hey, are you try to use calc model to detect loops in SLAM system? How can we combine the deep model and whole slam system?

100 landmarks

Hello, thank you for sharing your code. How did you get the 100 landmarks? First get some proposals using BING, and then generate the final detection using a detection? But the results of the final detection are some bounding boxes. I think landmarks should be some points rather than bounding boxes. Thank you very much.

compile DBoW2 error

Hi Nate
When I compiled DBoW2, I got the following error:
calc/TrainAndTest/ThirdParty/DBoW2/src/pywrap.cpp:125:17: error: ‘boost::python::numeric’ has not been declared
boost::python::numeric::array::set_module_and_type("numpy", "ndarray");

How should I solve it?

An error in TrainAndTest

Excuse me. When I run the script
python main.py test -m ../DeepLCD/calc_model/ -d test_data/CampusLoopDataset/
I get this error. Can you help me?

Run testNet.py

Hi Nate.
I have installed all the required dependencies and successfully compiled the DeepLCD library. Now I want to see the precision-recall curves(comparing against HOG and DBoW2), so I run the following command:
$ cd ~/calc/TrainAndTest
$ python testNet.py
Unfortunately, It is nothing output.
What commands should I run to print the precision-recall curves?

about training

hello,i read your paper and understand the train dataset is Place dataset, but which version of the dataset are you using for training? and the number of iterations of network training? The are two version.1.8T and the other is resized.I trained the net using the resized database version and used 400,000 iterations , but the experimental results obtained are very poor.Can you tell me what version are you using and the number of iterations of network training? Thand you!

Implementation about denoising auto encoder

Hello! I notice that denoising auto encoder is used as benchmark. And I am interested in the implementation of that AE. Could you provide the network definition file?

what is the dataset you use in the ros online demo

what is the pos_topic

When I execute cmake .. && make in the build folder of DeepLCD I get the following error

I installed everything laid out in the DeepLCD README.md and even tried installing gtest manually, but this persists.

Is the resolution of Places dataset you used 256*256?

Hi,
I use the Places dataset(256*256) to train the convolutional autoencoder, but the train loss decline from 130 to about 60, and the loss could not decline. Could u tell me what's wrong? Thank u so much

No Image received

Hello, i've been trying your code recently, but i ran into this problem with DeepLCD/online-demo_ws. Can you help me with this?

error: ‘accumulate’ is not a member of ‘std’

in ./src/speed-test/speed-test.cpp
there should inlcude the header of accumulate

#include <numeric>

deeplcd-test error

Hello,
i am on Ubuntu16.04 and build Caffe from src in CPU_ONLY mode.
i run ./deeplcd-test
and got this
...
I1107 10:07:56.879565 11560 net.cpp:202] relu1 does not need backward computation.
I1107 10:07:56.879568 11560 net.cpp:202] conv1 does not need backward computation.
I1107 10:07:56.879571 11560 net.cpp:202] X1 does not need backward computation.
I1107 10:07:56.879573 11560 net.cpp:244] This network produces output descriptor
I1107 10:07:56.879580 11560 net.cpp:257] Network initialization done.
I1107 10:07:56.899730 11560 net.cpp:746] Ignoring source layer X2
I1107 10:07:56.899821 11560 net.cpp:746] Ignoring source layer fc4
I1107 10:07:56.899827 11560 net.cpp:746] Ignoring source layer sig4
I1107 10:07:56.899828 11560 net.cpp:746] Ignoring source layer fc5
I1107 10:07:56.899830 11560 net.cpp:746] Ignoring source layer sig5
I1107 10:07:56.899832 11560 net.cpp:746] Ignoring source layer fc6
I1107 10:07:56.899834 11560 net.cpp:746] Ignoring source layer X2_comp_hat
I1107 10:07:56.899837 11560 net.cpp:746] Ignoring source layer Flatten1
I1107 10:07:56.899838 11560 net.cpp:746] Ignoring source layer loss

Caffe mode = CPU
Loaded model in 0.020969 seconds
[ OK ] DeepLCDTest.TestFullConstructor (22 ms)
[ RUN ] DeepLCDTest.TestAddMulti
[ OK ] DeepLCDTest.TestAddMulti (127 ms)
[ RUN ] DeepLCDTest.TestMultiImageRetrieval
[ OK ] DeepLCDTest.TestMultiImageRetrieval (127 ms)
[----------] 6 tests from DeepLCDTest (575 ms total)

[----------] Global test environment tear-down
[==========] 6 tests from 1 test case ran. (575 ms total)
[ PASSED ] 6 tests.
*** Error in `./deeplcd-test': free(): invalid next size (normal): 0x0000000002472f80 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ff48e6417e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7ff48e64a37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7ff48e64e53c]
/usr/lib/x86_64-linux-gnu/libopencv_ts.so.2.4(_ZNSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS5_EED1Ev+0x29)[0x7ff48fe72ad9]
/lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0x9a)[0x7ff48e60436a]
/usr/lib/x86_64-linux-gnu/libopencv_ts.so.2.4(+0x1f633)[0x7ff48fe72633]
This is only the last part of the log, the full log is attached.
Let me know if you have any suggestions.
Thanks and regards,
Pietro
log.txt

Cannot use GPU in CPU-only Caffe: check mode. *** Check failure stack trace: ***

Hi,
Just I want to test Calc, I type this cmd:
python main.py test
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0719 15:04:57.433012 24072 common.cpp:66] Cannot use GPU in CPU-only Caffe: check mode.
*** Check failure stack trace: ***
Did I operated wrong ? Any Help,Thanks

make error

Hello, how can I solve this problem?

question about the results?

Hello, thanks for sharing your work here~I have some problems about the results. First, I train the net from the scratch on the MIT places dataset (use train and val images) as you said in other issues. I also observed that the loss is about 60-65, it doesn't decay. After that, I test the model on the campus dataset(test data in your code) at training epoch 100000, 200000, 300000, 400000, 500000 and so on. But I can not get the similar result as your model. My model is just a little better than the HoG, I am so confused .... If the loss function doesn‘t mean when the network has been trained ok because the HoG is not good for this task. The weights in the network will change according to what? It seems reasonable that my model is just better than HoG according to the 60-65 loss. Secondly, I use Alexnet (trained on the MIT places dataset) in the your test code, it achieves different and better results in your test data, and I use GardensPoint and aderley dataset(the last 200 frames as you said in your paper), It also performs better , and I can not get the same result using the released calc model weight. Looking forward to your reply, thanks!

A standalone sample

Hi,
Can you suggest me how can I test your algorithm with my own sequence? So my sequence is a series of images very similar to gradens-point dataset. I wish to extract the scene descriptors from your method for each of the images and compare the loop detections.

implementation about the LA model

Hi, nate. Do you have the open source code for the model created by Lopez-Antequera ?

BGR2YUV and YUV2BGR?

In testNet.py --> def computeForwardPasses() there are following conversions:

BGR2YUV and YUV2BGR.

I am just wondering why we should have these two conversions and not instead use BGR2GRAY directly?

Cheers,

make error

Hi, nate. When compile:

make[2]: *** No rule to make target 'src/googletest-build/googlemock/gtest/libgtest.a', needed by 'deeplcd-test'。 停止。
CMakeFiles/Makefile2:211: recipe for target 'CMakeFiles/deeplcd-test.dir/all' failed
make[1]: *** [CMakeFiles/deeplcd-test.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2

Do you know how to deal with this issue??

Why HOG?

I am wondering if you ever tried other feature descriptors such as ORB or SIFT? Did they perform poorly for deep loop clousre?

What is the value of loss?

Hi @nmerrill67 ,
I'm finetuing calc network on our own dataset. Could you provide the final value of your training loss? Mine is 63, which I think is too large, and the PR Curve is also worse than your weights.
I trained the network 500k iterations.

Question about training on my own dataset

Hello, if I want to use your net model train on my dataset, what should I do?

gtest error

How to slove？

Using DeepLCD in LSD-SLAM

Hi, I have now embedded Deeplcd into LSD-SLAM for closed-loop detection. But when I run ROS, I get the error in the picture.

make error

[ 50%] Building CXX object CMakeFiles/deeplcd.dir/src/deeplcd/deeplcd.cpp.o
In file included from /home/ydy/calc/DeepLCD/src/deeplcd/deeplcd.cpp:1:0:
/home/ydy/calc/DeepLCD/include/deeplcd/deeplcd.h:1:27: fatal error: caffe/caffe.hpp: no such file or dictionary
compilation terminated.
CMakeFiles/deeplcd.dir/build.make:62: recipe for target 'CMakeFiles/deeplcd.dir/src/deeplcd/deeplcd.cpp.o' failed
make[2]: *** [CMakeFiles/deeplcd.dir/src/deeplcd/deeplcd.cpp.o] Error 1
CMakeFiles/Makefile2:136: recipe for target 'CMakeFiles/deeplcd.dir/all' failed
make[1]: *** [CMakeFiles/deeplcd.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2

I am using Ubuntu16.04 and I'm sure I install caffe under /home/usr. I don't know Y it can not find caffe.hpp, it acually exist under /home/usr/caffe/include/caffe

landmark