prbonn / pin_slam

📍PIN-SLAM: LiDAR SLAM Using a Point-Based Implicit Neural Representation for Achieving Global Map Consistency [TRO' 24]

📍PIN-SLAM: LiDAR SLAM Using a Point-Based Implicit Neural Representation for Achieving Global Map Consistency

Yue Pan · Xingguang Zhong · Louis Wiesmann . Thorbjörn Posewsky . Jens Behley · Cyrill Stachniss

University of Bonn

Paper | Video

TL;DR: PIN-SLAM is a full-fledged implicit neural LiDAR SLAM system including odometry, loop closure detection, and globally consistent mapping

Globally consistent point-based implicit neural (PIN) map built with PIN-SLAM in Bonn. The high-fidelity mesh can be reconstructed from the neural point map.

---

Comparison of (a) the inconsistent mesh with duplicated structures reconstructed by PIN LiDAR odometry, and (b) the globally consistent mesh reconstructed by PIN-SLAM.

---

Globally Consistent Mapping	Various Scenarios	RGB-D SLAM Extension
demo_kitti00.mp4	demo_lidar_9scenes.mp4	demo_replica_rgbd.mp4

Table of Contents

1. Abstract
2. Installation
3. How to run PIN-SLAM
4. Visualizer instructions
5. Citation
6. Contact
7. Related projects

Abstract

[Details (click to expand)]

Accurate and robust localization and mapping are essential components for most autonomous robots. In this paper, we propose a SLAM system for building globally consistent maps, called PIN-SLAM, that is based on an elastic and compact point-based implicit neural map representation. Taking range measurements as input, our approach alternates between incremental learning of the local implicit signed distance field and the pose estimation given the current local map using a correspondence-free, point-to-implicit model registration. Our implicit map is based on sparse optimizable neural points, which are inherently elastic and deformable with the global pose adjustment when closing a loop. Loops are also detected using the neural point features. Extensive experiments validate that PIN-SLAM is robust to various environments and versatile to different range sensors such as LiDAR and RGB-D cameras. PIN-SLAM achieves pose estimation accuracy better or on par with the state-of-the-art LiDAR odometry or SLAM systems and outperforms the recent neural implicit SLAM approaches while maintaining a more consistent, and highly compact implicit map that can be reconstructed as accurate and complete meshes. Finally, thanks to the voxel hashing for efficient neural points indexing and the fast implicit map-based registration without closest point association, PIN-SLAM can run at the sensor frame rate on a moderate GPU.

Installation

Platform requirement

• Ubuntu OS (tested on 20.04)

• With GPU (recommended) or CPU only (run much slower)

• GPU memory requirement (> 6 GB recommended)

• Windows/MacOS with CPU-only mode

1. Set up conda environment

conda create --name pin python=3.8
conda activate pin

2. Install the key requirement PyTorch

conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.7 -c pytorch -c nvidia 

The commands depend on your CUDA version. You may check the instructions here.

3. Install other dependency

pip3 install -r requirements.txt

---

Run PIN-SLAM

Clone the repository

git clone [email protected]:PRBonn/PIN_SLAM.git
cd PIN_SLAM

Sanity test

For a sanity test, do the following to download an example part (first 100 frames) of the KITTI dataset (seq 00):

sh ./scripts/download_kitti_example.sh

And then run:

python3 pin_slam.py ./config/lidar_slam/run_demo.yaml -vsm

[Details (click to expand)]

You can visualize the SLAM process in PIN-SLAM visualizer and check the results in the ./experiments folder.

Use run_demo_sem.yaml if you want to conduct metric-semantic SLAM using semantic segmentation labels:

python3 pin_slam.py ./config/lidar_slam/run_demo_sem.yaml -vsm

If you are running on a server without an X service (you may first try export DISPLAY=:0), then you can turn off the visualization -v flag:

python3 pin_slam.py ./config/lidar_slam/run_demo.yaml -sm

If you don't have a Nvidia GPU on your device, then you can turn on the CPU-only operation by adding the -c flag:

python3 pin_slam.py ./config/lidar_slam/run_demo.yaml -vsmc

Run on your datasets

For an arbitrary data sequence, you can run with the default config file by:

python3 pin_slam.py -i /path/to/your/point/cloud/folder -vsm

[Details (click to expand)]

Follow the instructions on how to run PIN-SLAM by typing:

python3 pin_slam.py -h

To run PIN-SLAM with a specific config file, you can run:

python3 pin_slam.py path_to_your_config_file.yaml -vsm

The flags -v, -s, -m toggle the visualizer, map saving and mesh saving, respectively.

To specify the path to the input point cloud folder, you can either set pc_path in the config file or set -i INPUT_PATH upon running.

For pose estimation evaluation, you may also set pose_path in the config file to specify the path to the reference pose file (in KITTI or TUM format).

For some popular datasets, you can also set the dataset name and sequence name upon running. For example:

# KITTI dataset sequence 00
python3 pin_slam.py ./config/lidar_slam/run_kitti.yaml kitti 00 -vsm

# MulRAN dataset sequence KAIST01
python3 pin_slam.py ./config/lidar_slam/run_mulran.yaml mulran kaist01 -vsm

# Newer College dataset sequence 01_short
python3 pin_slam.py ./config/lidar_slam/run_ncd.yaml ncd 01 -vsm

# Replica dataset sequence room0
python3 pin_slam.py ./config/rgbd_slam/run_replica.yaml replica room0 -vsm

We also support loading data from rosbag, mcap or pcap using specific data loaders (originally from KISS-ICP). You need to set the flag -d to use such data loaders. For example:

# Run on a rosbag or a folder of rosbags with certain point cloud topic
python3 pin_slam.py ./config/lidar_slam/run.yaml rosbag point_cloud_topic_name -i /path/to/the/rosbag -vsmd

# If there's only one topic for point cloud in the rosbag, you can omit it
python3 pin_slam.py ./config/lidar_slam/run.yaml rosbag -i /path/to/the/rosbag -vsmd

The data loaders for some specific datasets are also available.

Available dataloaders: ['apollo', 'boreas', 'generic', 'helipr', 'kitti', 'kitti_raw', 'mcap', 'mulran', 'ncd', 'nclt', 'neuralrgbd', 'nuscenes', 'ouster', 'paris_luco', 'replica', 'rosbag', 'tum']

For example, you can run on Replica RGB-D dataset without preprocessing the data by:

# Download data
sh scripts/download_replica.sh

# Run PIN-SLAM
python3 pin_slam.py ./config/rgbd_slam/run_replica.yaml replica room0 -i data/Replica -vsmd 

The SLAM results and logs will be output in the output_root folder set in the config file or specified by the -o OUTPUT_PATH flag.

For evaluation, you may check here for the results that can be obtained with this repository on a couple of popular datasets.

The training logs can be monitored via Weights & Bias online if you set the flag -w. If it's your first time using Weights & Bias, you will be requested to register and log in to your wandb account. You can also set the flag -l to turn on the log printing in the terminal and set the flag -r to turn on the visualization logging by rerun.

ROS 1 Support

If you are not using PIN-SLAM as a part of a ROS package, you can avoid the catkin stuff and simply run:

python3 pin_slam_ros.py path_to_your_config_file.yaml point_cloud_topic_name

[Details (click to expand)]

For example:

python3 pin_slam_ros.py ./config/lidar_slam/run.yaml /os_cloud_node/points

After playing the ROS bag or launching the sensor you can then visualize the results in Rviz by:

rviz -d ./config/pin_slam_ros.rviz 

You may use the ROS service save_results and save_mesh to save the results and mesh (at a default resolution) in the output_root folder.

rosservice call /pin_slam/save_results
rosservice call /pin_slam/save_mesh

The process will stop and the results and logs will be saved in the output_root folder if no new messages are received for more than 30 seconds.

If you are running without a powerful GPU, PIN-SLAM may not run at the sensor frame rate. You need to play the rosbag with a lower rate to run PIN-SLAM properly.

You can also put pin_slam_ros.py into a ROS package for rosrun or roslaunch.

We will add support for ROS2 in the near future.

Inspect the results after SLAM

After the SLAM process, you can reconstruct mesh from the PIN map within an arbitrary bounding box with an arbitrary resolution by running:

python3 vis_pin_map.py path/to/your/result/folder [marching_cubes_resolution_m] [(cropped)_map_file.ply] [output_mesh_file.ply] [mesh_min_nn]

[Details (click to expand)]

The bounding box of (cropped)_map_file.ply will be used as the bounding box for mesh reconstruction. This file should be stored in the map subfolder of the result folder. You may directly use the original neural_points.ply or crop the neural points in software such as CloudCompare. The argument mesh_min_nn controls the trade-off between completeness and accuracy. The smaller number (for example 6) will lead to a more complete mesh with more guessed artifacts. The larger number (for example 15) will lead to a less complete but more accurate mesh. The reconstructed mesh would be saved as output_mesh_file.ply in the mesh subfolder of the result folder.

For example, for the case of the sanity test described above, run:

python3 vis_pin_map.py ./experiments/sanity_test_*  0.2 neural_points.ply mesh_20cm.ply 8

Visualizer Instructions

We provide a PIN-SLAM visualizer based on lidar-visualizer to monitor the SLAM process. You can use -v flag to turn on it.

[Keyboard callbacks (click to expand)]

Button	Function
Space	pause/resume
ESC/Q	exit
G	switch between the global/local map visualization
E	switch between the ego/map viewpoint
F	toggle on/off the current point cloud visualization
M	toggle on/off the mesh visualization
A	toggle on/off the current frame axis & sensor model visualization
P	toggle on/off the neural points map visualization
D	toggle on/off the training data pool visualization
I	toggle on/off the SDF horizontal slice visualization
T	toggle on/off PIN SLAM trajectory visualization
Y	toggle on/off the ground truth trajectory visualization
U	toggle on/off PIN odometry trajectory visualization
R	re-center the view point
Z	3D screenshot, save the currently visualized entities in the log folder
B	toggle on/off back face rendering
W	toggle on/off mesh wireframe
Ctrl+9	Set mesh color as normal direction
5	switch between point cloud for mapping and for registration (with point-wise weight)
7	switch between black and white background
/	switch among different neural point color mode, 0: geometric feature, 1: color feature, 2: timestamp, 3: stability, 4: random
<	decrease mesh nearest neighbor threshold (more complete and more artifacts)
>	increase mesh nearest neighbor threshold (less complete but more accurate)
[/]	decrease/increase mesh marching cubes voxel size
↑/↓	move up/down the horizontal SDF slice
+/-	increase/decrease point size

Citation

If you use PIN-SLAM for any academic work, please cite our original paper.

@article{pan2024tro,
author = {Y. Pan and X. Zhong and L. Wiesmann and T. Posewsky and J. Behley and C. Stachniss},
title = {{PIN-SLAM: LiDAR SLAM Using a Point-Based Implicit Neural Representation for Achieving Global Map Consistency}},
journal = IEEE Transactions on Robotics (TRO),
year = {2024},
codeurl = {https://github.com/PRBonn/PIN_SLAM},
}

Contact

If you have any questions, please contact:

• Yue Pan {[email protected]}

Related Projects

SHINE-Mapping (ICRA 23): Large-Scale 3D Mapping Using Sparse Hierarchical Implicit Neural Representations

LocNDF (RAL 23): Neural Distance Field Mapping for Robot Localization

KISS-ICP (RAL 23): A LiDAR odometry pipeline that just works

4DNDF (CVPR 24): 3D LiDAR Mapping in Dynamic Environments using a 4D Implicit Neural Representation

pin_slam's Issues

odometry evaluation

Thanks for your excellent work!!!
I run this codes in kitti_04,and get absolute trajectory error 0.34 and your paper is 0.1 in table IV. So how can i get the same result?

Have the values of `local_neural_points` been modified between being selected from and reassigned to global neural points?

Thanks for your great work! I have the following question:

In NeuralPoints.reset_neural_points(), local_neural_points are selected from global neural points:

self.local_neural_points = self.neural_points[local_mask]

In NeuralPoints.assign_local_to_global(), local_neural_points are reassigned to global neural points:

self.neural_points[local_mask] = self.local_neural_points

But after searching all the usages of local_neural_points, I cannot see whether their values are modified anywhere in the code. In other words, doself.neural_points[local_mask] remain the same values after the NeuralPoints.assign_local_to_global() operation?

Thanks for your time and explanation.

Z Drift in custom dataset

Thanks a lot for this great work!

I meet some problems when I test pin_slam.

When I test in my custom data, I found there will have large z drift (about 8-9m) in long term(5-6km) travel. and the loop detector will be failed.
I tried to adjust some parameters in pgo and loop detector but got no improve.

Is there any idea to solve this, I will try to count distance only use x,y and ignore z for loop detection, but I dont think this is a good solution.

thanks again.

How to understand the stability of neural points?

Hi @YuePanEdward

Thanks for your excellent work and great open-sourced code!
I find that It is kind of difficult for me to understand the stability of neural points in the paper.
It seems that the larger the stability value, the less reliable the neural points are, and the easier nearby query points are to be eliminated.
But the stability value looks like just the sum of the weights given when the neural points are queried.
I am not sure my understanding is right.
How should we understand this stability intuitively?

poor result on new college dataset.

Hi, Thanks for your excellent work !!!
I want to know how to get better result as your paper in TABLE XI on Quad. I use the default config file: ./config/Lidar_slam/run_ncd.yaml. i generate mesh use command " python vis_pin_map.py ./experiments/ncd_**** 0.2 nueral_points.ply mesh_20cm.ply". and here is my result:

Predicted mesh unifrom sample: 10000000 --> 8563620 ( 0.02 m)
{'MAE_accuracy (m)': 0.16234044696769342, 'MAE_completeness (m)': 0.1927699826745742, 'Chamfer_L1 (m)': 0.17755521482113382, 'Chamfer_L2 (m)': 0.23996385734393333, 'Precision [Accuracy] (%)': 64.42148915027659, 'Recall [Completeness] (%)': 62.22490167539342, 'F-score (%)': 63.304146324647974, 'Spacing (m)': 0.02, 'Inlier_threshold (m)': 0.2, 'Outlier_truncation_acc (m)': 0.4, 'Outlier_truncation_com (m)': 2.0}

could you please tell me how can get the same result as you.
here are the mesh and my config file screenshot:

Unexpected artifacts when only perform mapping with gt pose.

Hi @YuePanEdward Thanks for your great work and making it open source!

I have met a strange problem when I run the mapping process on KITTI seq07 with tracking module disabled.
I straightforward use groundtruth poses for mapping and use run_kitti.yaml as config file.
Then I noticed that the reconstruction of this region is not satisfactory when using ground truth poses, but if I use the full SLAM system with tracking, this region is nicely reconstructed.

This problem is kind of wired. I would appreciate it if you could provide some advice and hints on how to figure it out.
One potential reason is the car stopped here for a while, so I haved tried to remove these duplicated frames but it seems not helps.

question about parameter correct_deg:0.195 in run_kitti.yaml

Thanks for your excellent work!!!!!!!!!
For kitti dataset, there is a parameter named correct_deg 0.195. I wanna ask what is the purpose of this parameter, and why is this parameter 0.195?

How many GPU memory need for generate a full scene mesh video like [demo_kitti00.mp4 ] ?

How to setup config file?

CUDA error even when using cpu yaml

Hello and thank you for your great work.

I was trying to use this command python pin_slam.py ./config/lidar_slam/run_demo_cpu.yaml to run the initial test but i get this error:

  File "pin_slam.py", line 380, in <module>
    run_pin_slam()
  File "pin_slam.py", line 103, in run_pin_slam
    T0 = get_time()
  File "/home/omar-nour/catkin_ws/src/PIN_SLAM/utils/tools.py", line 273, in get_time
    torch.cuda.synchronize()
  File "/home/omar-nour/miniconda3/envs/pin/lib/python3.8/site-packages/torch/cuda/__init__.py", line 799, in synchronize
    _lazy_init()
  File "/home/omar-nour/miniconda3/envs/pin/lib/python3.8/site-packages/torch/cuda/__init__.py", line 302, in _lazy_init
    torch._C._cuda_init()
RuntimeError: No CUDA GPUs are available

I am trying to run PIN-SLAM on Ubuntu 20.04 Virtual Machine (Virtual Box) with ROS Noetic

issue about mesh

Thank you for your outstanding work. I have a question about the kitti00 dataset. I observed that there are some blank areas behind the car. Is this because the radar did not detect the point clouds at the rear?

About time consumption

Thank you for your wonderful work！

I run your code on GPU Titan V , 12 GB and
set the pool_capacity: 5e6, batch_size:8192;
But I don't know why my time consumption is so long.
On Kitti-00, running time for 43 min.
I want to ask how to adapt it to my GPU for real-time operation;

this my time details on Kitti-00:

sometime odometry time consumption over 200ms:

Thank you for your help!

Have individual data loader for each dataset

It would be better to have a separate data loader for each dataset. In this case, we can avoid converting all the datasets to the desired format (point cloud in kitti bin, ply, pcd format and pose in kitti format). We can also better handle the loading of point-wise timestamps.

Would the training code be released?

Hi, I am very interested in your work. May I know if you plan to release the training code?

torch.cuda.OutOfMemoryError: CUDA out of memory

Hello, when I try to run your code run_demo.yaml，the following error has occurred:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 396.00 MiB. GPU 0 has a total capacity of 3.81 GiB of which 348.00 MiB is free.
My gpu is RTX3050,How can I modify this program to make it run properly.
Thank you!

LiDAR points and Visual RGB

Thank you for your wonderful work！

1. How to integrate LiDAR point clouds with visual RGB?
2. Is there a recommended dataset?

Thank you!

grid_idx

grid_idx = grid[:, 0] + grid[:, 1] * v_size + grid[:, 2] * v_size * v_size
Will this conversion method not be repeated？

Noisy Mesh

Thanks for the incredible open-source project! I was running PIN-SLAM with the provided run_demo yaml file and the 100-frame KITTI from the download script, and got a quite noisy mesh. Meshes from the paper and video look really smooth, so I'm wondering what should I do to address this? Thanks.

My GPU is RTX 4080 if it helps.

Attitude drift after making a turn in custom dataset

Top(at time 1):

Side(at time 1):

Side(at time 2):

It will make a z drift and increase unbound

Difficult to operate at the sensor frequency in kitti 00

Hi~, thanks for your great work and the open source. We tested on a RTX3080Ti GPU with 12 GB memory. The average consuming time per frame is 245.8ms in the whole KITTI 00 dataset. I think the performance of RTX3080Ti is similar to the A4000. The parameters are all default. Can you give us some advice? Thank you!

Clarify dependencies / requirements

Hi,

First of all, thanks for the great work and for open-sourcing it! I am really curious about trying it out.

When following the installation instructions on Ubuntu 20.04, I get the following error:

Traceback (most recent call last):
  File "pin_slam.py", line 11, in <module>
    import open3d as o3d
  File "/home/jhpauls/anaconda3/envs/pin_slam/lib/python3.8/site-packages/open3d/__init__.py", line 146, in <module>
    import open3d.ml
  File "/home/jhpauls/anaconda3/envs/pin_slam/lib/python3.8/site-packages/open3d/ml/__init__.py", line 16, in <module>
    from . import datasets
  File "/home/jhpauls/anaconda3/envs/pin_slam/lib/python3.8/site-packages/open3d/ml/datasets.py", line 15, in <module>
    from open3d._ml3d.datasets import *
  File "/home/jhpauls/anaconda3/envs/pin_slam/lib/python3.8/site-packages/open3d/_ml3d/datasets/__init__.py", line 3, in <module>
    from .semantickitti import SemanticKITTI
  File "/home/jhpauls/anaconda3/envs/pin_slam/lib/python3.8/site-packages/open3d/_ml3d/datasets/semantickitti.py", line 6, in <module>
    from sklearn.neighbors import KDTree
  File "/home/jhpauls/anaconda3/envs/pin_slam/lib/python3.8/site-packages/sklearn/__init__.py", line 83, in <module>
    from .base import clone
  File "/home/jhpauls/anaconda3/envs/pin_slam/lib/python3.8/site-packages/sklearn/base.py", line 19, in <module>
    from .utils import _IS_32BIT
  File "/home/jhpauls/anaconda3/envs/pin_slam/lib/python3.8/site-packages/sklearn/utils/__init__.py", line 15, in <module>
    from scipy.sparse import issparse
  File "/home/jhpauls/.local/lib/python3.8/site-packages/scipy/__init__.py", line 70, in <module>
    from ._lib.deprecation import _deprecated
ModuleNotFoundError: No module named 'scipy._lib'

I suspect the default versions of e.g. scipy and/or sklearn have changed.

Could you export the exact dependencies in a requirements / environment file or specify them in the README?

Thanks a lot!

Transformation matrix

Hello,
Thanks for the great work and for open-sourcing it.
I want to use another method for registration. When I change the T_mat matrix in the Tracker.py file (line 666) and I use the transformation matrix of my own method, I don't get a proper output.
Can you tell me which other parts of the code I need to change so that I can apply my own transformation matrix?

Add pure-localization mode

It would be nice to have a pure-localization mode so that we can localize the robot inside a pre-built map. This may also work for the revisited region. We can get rid of the repeated mapping and only do the localization when there are not much new observations.

signal frame

Thank you for your work. I would like to rebuild a mesh with point clouds for each of my frames and save it. Is this project feasible?
Mesh should be generated based on the completed sequence, rather than directly using a single frame point cloud to generate mesh？

How to use Replica dataset

About the VBR dataset

Hi, thanks for your impressive work. I see there is a run_vbr.yaml in config folder. But after I download the VBR dataset, there is no /ply folder, so I just want to ask, if you do some convert processing on vbr dataset, using like scripts/rosbag2ply.py? Additionally in each scene in vbr, there are several .bag, how do you combine them?

Thanks for your early reply.