kxhit / sg_pr

Implementation of IROS20 paper - "Semantic Graph Based Place Recognition for 3D Point Clouds"

SG_PR

Code for IROS2020 paper Semantic Graph Based Place Recognition for 3D Point Clouds

Pipeline overview.

Citation

If you think this work is useful for your research, please consider citing:

@inproceedings{kong2020semantic,
  title={Semantic Graph based Place Recognition for Point Clouds},
  author={Kong, Xin and Yang, Xuemeng and Zhai, Guangyao and Zhao, Xiangrui and Zeng, Xianfang and Wang, Mengmeng and Liu, Yong and Li, Wanlong and Wen, Feng},
  booktitle={2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages={8216--8223},
  year={2020},
  organization={IEEE}
}

Requirements

We recommend python3.6. You can install required dependencies by:

pip install -r requirements.txt

Training

Data structure

The data structure is:

data
    |---00
    |    |---000000.json   
    |    |---000001.json
    |    |---...
    |
    |---01
    |    |---000000.json
    |    |---000001.json
    |    |---...
    |
    |---...
    |
    |---00.txt
    |---01.txt
    |---...

You can download the provided preprocessed data. Or you can refer to the 'data_process' dir for details of generating graphs.

Configuration file

Before training the model, you need to modify the configuration file in ./config according to your needs. The main parameters are as follows:

• graph_pairs_dir: the root dir of your dataset.
• batch_size: batch size, 128 in our paper.
• p_thresh: distance threshold for positive samples, e.g., 3m 5m 10m. If the distance between two samples is less than p_thresh meters, they will be treated as positive samples. The distance threshold for negative samples is set to 20 meters by default. Note that your training sample pairs should not contain samples with a distance greater than p_thresh meters and less than 20 meters.

Training model

• model: pre-trained model.
• train_sequences: list of training data.
• eval_sequences: list of validation data.
• logdir: path to save training results.
• graph_pairs_dir: set to SK label dir, e.g., '../SG_PR_DATA/graphs_sk'
• pair_list_dir: set to train dir, e.g., '../SG_PR_DATA/train/3_20', 3_20 refers to positive threshold 3m negative threshold 20m

After preparing the data and modifying the configuration file, you can start training. Just run:

python main_sg.py

Testing

eval_pair

This example takes a pair of graphs as input and output their similarity score. To run this example, you need to set the following parameters:

• model: the eval model file.
• pair_file: a pair of graph.

Then just run:

python eval_pair.py

eval_batch

This example tests a sequence, the results are it's PR curve and F1 max score. To run this example, you need to set the following parameters:

• model: the eval model file.
• graph_pairs_dir: set to SK label dir ('../SG_PR_DATA/graphs_sk') or RN prediction dir ('../SG_PR_DATA/graphs_rn') or other semantic prediction in the future.
• pair_list_dir: set to eval dir which excludes easy positive pairs, e.g., '../SG_PR_DATA/eval/3_20'
• sequences: list of test sequences.
• output_path: path to save test results.
• show: whether to display the pr curve in real time.

Then just run:

python eval_batch.py

Raw Data

We provide the raw data of the tables and curves in the paper, including compared methods M2DP, PointNetVLAD, Scan Context.

We recommend users refer the work SSC for a fair comparison with recent methods in the same data distribution.

Other methods

PointNetVLAD

Please refer to our modified repo for training and testing PointNetVLAD on KITTI dataset, which is mentioned in our paper as PV_KITTI.

TODO

• Support Ford Campus Dataset
• Release compared methods e.g., PointNetVLAD trained on KITTI (PV-KITTI)
• Release preprocessing code

Acknowledgement

Thanks to the source code of some great works such as SIMGNN, DGCNN.

sg_pr's Issues

Time for training

Hi！I want to know the time required for the network training, and is epochs 500?

distance error:

I test pair_file: ["/SG_PR_DATA/graphs_sk/05/000000.json","/SG_PR_DATA/graphs_sk/05/000006.json"] in eval_pair.py.
The rusult score more than 1.
------------------------------------------------------------------------------console output:
Enumerating unique labels.

{0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11}
12
loading model: ./model/model.pth
distance error: 3.3904110318197205

I,m confused.

about the testing and training

Thanks for your great contribution!
I have a question for testing and training part

You described "We use 1-fold cross-validation and
each sequence is considered as a fold, that is, consider one
sequence as a test set and the others as training sets"

And I understood the LoopClosureDetection based 00,02,05,07,08 sequnce

Could you please tell me in details about which sequnce you used for the 1fold cross validation for testing and training?

Hoping for your reply.
Thanks a lot.

2 bugs about gpu_id

sg_net.py using "cuda:0"
dgcnn.py using "cuda"
your could exchange it with "cuda:"+str(args.gpu_id)

How to visualize the Fig.7

Congratulations, author, great work. I really want to know how to visualize Fig. 7 in this paper? Do you have sample code?

Training

Hello, I have used your semantic mapping solution, starting with pairing to create data. However, the F1 score on my first training attempt was 0.9. May I know what could be the reason? I have replaced the original clustering method.

About Comparative Experiment

Hello, I would like to ask if the data set of the comparative experiment (such as pointnetvlad) in the paper is also divided according to your script？The pair_list I generated according to this script and I got f1_ score seems that the lowest score is about 0.65， It should not be lower than 0.129 in your paper.

dataset

Hello, I want to know how to run your code on the kitti dataset (referring to SG_PR). I hope to give more detailed steps in the readme document. I look forward to your answer, thank you.

about train

how to generate text file(like 00.txt) of train?

Ask questions about data usage

Hello，I have benefited a lot from reading your code. How to use the data in rawdata, such as 00_M2DP_db.npy and 00_gt_db.npy in the M2DP00 folder

Missing code

The code is missing

A question about the code in sg_net.py

Hi, thanks for your interesting work! I'm going deep into the code and have a question in line 326 of sg_net.py. The code is as below. In my opinion, batch_feature_1 should be the concatenated feature of one frame. So as batch_feature_2. For what reason makes you append data["features_2"] into batch_feature_1? Tell me if I miss any point. Thanks again for your reply.
for graph_pair in batch:
data = process_pair(graph_pair)
data = self.transfer_to_torch(data, training)
batch_feature_1.append(data["features_1"])
batch_feature_2.append(data["features_2"])
batch_feature_1.append(data["features_2"])
batch_feature_2.append(data["features_1"])
target = data["target"]
batch_target.append(target)
batch_target.append(target)

Dalao, how to get graphs_rn and graphs_sk with rangenet++ and Kitti dataset ?

about how to get pre-processing data format

Hello, I'd like to ask how i can get the JSON file format you published in advance. For example, there is pose data in the JSON data you published in advance. You did use pose later, but the data preprocessing code does not involve how to calculate pose ? Can you help me answer my question.Thank you.

about the pre-processing

Hi, I have a question about the pre-processing. In the paper, you process the raw point cloud by segmentation and ignore classes like person, because they are either irrelevant or few in number. So why not ignore the vehicle class, which I think is movable and would interfere with place recognition.

How should I modify the training params to achieve the accuracy in your paper?

Hi, I am retraining your model with default settings on a GTX1080 GPU, except the batch size is set to 256, and the nodes used are from semantic Kitti presented by you. When I first test sequence 08, which means sequences 00-07,09,10 as the training sets, the F1 max should achieve 0.900 if I wasn't missing any point. However, the best training result is only around 0.66(F1 max). Could you please give some suggestions on this situation?

Codes wanted

I have read your paper in IROS 2020. It is an amazing work!
Could you share the code as soon as possible? I can't wait to reproduce your job now.

Thanks a lot!

pair_list issue

Hello, this work is quite amazing and most of the documentation is very clear. But I am a little confused about how to generate the pair_list in the train folder in the preprocessed data you have provided.

about the classification part

Thanks for the great contribution of this research.
About the visualization part for the dir(data_process),Could you please tell me how could i visualize the .json format 3d data?
by the way,the processed data was classfied into 10 classes for RANGENET++,I would like to know the standard of the classification.
Thanks a lot.

How can I use multiple GPUs for training

Thank you very much for your open source work，but I do know how to use multiple to train this net，Orz，Please help me T_T