This repository is for 2DPASS introduced in the following paper
Xu Yan*, Jiantao Gao*, Chaoda Zheng*, Chao Zheng, Ruimao Zhang, Shuguang Cui, Zhen Li*, "2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds", ECCV 2022 [arxiv].
If you find our work useful in your research, please consider citing:
@InProceedings{yan20222dpass,
title={2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds},
author={Xu Yan and Jiantao Gao and Chaoda Zheng and Chao Zheng and Ruimao Zhang and Shuguang Cui and Zhen Li},
year={2022},
booktitle={ECCV}
}
@InProceedings{yan2022let,
title={Let Images Give You More: Point Cloud Cross-Modal Training for Shape Analysis},
author={Xu Yan and Heshen Zhan and Chaoda Zheng and Jiantao Gao and Ruimao Zhang and Shuguang Cui and Zhen Li},
year={2022},
booktitle={NeurIPS}
}
- 2022-10-11 Our new work for cross-modal knowledge distillation is accepted at NeurIPS 2022:smiley: (here).
- 2022-09-20 We release codes for SemanticKITTI single-scan and NuScenes π!
- 2022-07-03 2DPASS is accepted at ECCV 2022 π₯!
- 2022-03-08 We achieve 1st place in both single and multi-scans of SemanticKITTI and 3rd place on NuScenes-lidarseg π₯!
- pytorch >= 1.8
- yaml
- easydict
- pyquaternion
- lightning (tested with pytorch_lightning==1.3.8 and torchmetrics==0.5)
- torch-scatter (pip install torch-scatter -f https://data.pyg.org/whl/torch-1.9.0+${CUDA}.html)
- nuScenes-devkit (optional for nuScenes)
- spconv (tested with spconv==2.1.16 and cuda==11.1, pip install spconv-cu111==2.1.16)
Please download the files from the SemanticKITTI website and additionally the color data from the Kitti Odometry website. Extract everything into the same folder.
./dataset/
βββ
βββ ...
βββ SemanticKitti/
βββsequences
βββ 00/
β βββ velodyne/
| | βββ 000000.bin
| | βββ 000001.bin
| | βββ ...
β βββ labels/
| | βββ 000000.label
| | βββ 000001.label
| | βββ ...
| βββ image_2/
| | βββ 000000.png
| | βββ 000001.png
| | βββ ...
| calib.txt
βββ 08/ # for validation
βββ 11/ # 11-21 for testing
βββ 21/
βββ ...
Please download the Full dataset (v1.0) from the NuScenes website with lidarseg and extract it.
./dataset/
βββ
βββ ...
βββ nuscenes/
βββv1.0-trainval
βββv1.0-test
βββsamples
βββsweeps
βββmaps
βββlidarseg
You can run the training with
cd <root dir of this repo>
python main.py --log_dir 2DPASS_semkitti --config config/2DPASS-semantickitti.yaml --gpu 0
The output will be written to logs/SemanticKITTI/2DPASS_semkitti
by default.
cd <root dir of this repo>
python main.py --log_dir 2DPASS_nusc --config config/2DPASS-nuscenese.yaml --gpu 0 1 2
We take SemanticKITTI as an example.
cd <root dir of this repo>
python main.py --log_dir baseline_semkitti --config config/2DPASS-semantickitti.yaml --gpu 0 --baseline_only
You can run the testing with
cd <root dir of this repo>
python main.py --config config/2DPASS-semantickitti.yaml --gpu 0 --test --num_vote 12 --checkpoint <dir for the pytorch checkpoint>
Here, num_vote
is the number of views for the test-time-augmentation (TTA). We set this value to 12 as default (on a Tesla-V100 GPU), and if you use other GPUs with smaller memory, you can choose a smaller value. num_vote=1
denotes there is no TTA used, and will cause about ~2% performance drop.
You can download the models with the scores below from this Google drive folder.
Model (validation) | Reported (mIoU) | Reproduced (mIoU) | Parameters |
---|---|---|---|
2DPASS-SemanticKITTI | 69.3% | 70.0% | 1.9M |
2DPASS-NuScenes | 79.4% | 79.5% | 45.6M |
Here, we train the model for SemanticKITTI with more epochs and thus gain the higher mIoU. Note that the results on benchmarks are gained by training with additional validation set and using instance-level augmentation.
Code is built based on SPVNAS, Cylinder3D, xMUDA and SPCONV.
This repository is released under MIT License (see LICENSE file for details).