choyingw / scadc-depthcompletion

ICASSP 2021: Scene Completeness-Aware Lidar Depth Completion for Driving Scenario

SCADC-DepthCompletion

Scene Completeness-Aware Lidar Depth Completion for Driving Scenario, ICASSP 2021

Cho-Ying Wu and Ulrich Neumann, University of Southern California

The full example video link is here https://www.youtube.com/watch?v=FQDTdpMPKxs

Paper: https://arxiv.org/abs/2003.06945

Project page: https://choyingw.github.io/works/SCADC/index.html

Advantages:

👍 First research to attend scene-completeness in depth completion

👍 Sensor Fusion for lidar and stereo cameras

👍 Structured upper scene depth

👍 Precise lower scene

Prerequisite

Ubuntu 16.04/ 20.04
Python 3
PyTorch 1.5+ (Tested on 1.5, should be compatiable for following versions)
NVIDIA GPU + CUDA CuDNN 
Other common libraries: matplotlib, cv2, PIL

Data Preparation

Clone the repo first.

Then, download preprocessed data from train (142G) val (11G). This data includes training/val split that follows KITTI Completion and all required pre-processed data for this work.

Extract the files under the repository. The structure should be like 'SCADC-DepthCompletion/Data/train' and 'SCADC-DepthCompletion/Data/val'

*.h5 files are provided, including sparse depth (D), semi-dense depth (D_semi), left-right pairs (I_L and I_R), depth completed from SSDC (depth_c), and disparity from PSMNet (disp_c).

Evaluation/Training Commands:

Our provided pretrained weight is under './test_ckpt/kitti/'. To quickly get our scene completeness-aware depth maps, just use the evaluation command, and it will save frame-by-frame results under './vis/'. Download "val" data split in the Data Preparation section and unzip under 'data/'. The folder structure and the evaluation command should be

  .
  ├── data
        ├── val
           ├── 0
               ├── 00000.h5
	     ......
	     
python3 evaluate.py --name kitti --checkpoints_dir './test_ckpt' --test_path ./data

This is the training command is you want ot train the network yourself.

python3 train_depth_complete.py --name kitti --checkpoints_dir [preferred saving ckpt path] --train_path [train_data_dir] --test_path [test_data_dir]

[train_data_dir]: it should be 'Data/train' when you follow the recommended folder structure [test_data_dir]: it should be 'Data/test' when you follow the recommended folder structure

Customized depth completion and stereo estimation base methods:

Note that we use SSDC, and disparity from PSMNet.

The pre-processed data is in the *.h5 files. (key: 'depth_c' and 'disp_c'). If you want to make completion results from different basic methods, please prepare those data at your own and replace data stored in *.h5 files.

If you find our work useful, please consider to cite our work.

@inproceedings{wu2021scene,
  title={Scene Completeness-Aware Lidar Depth Completion for Driving Scenario},
  author={Wu, Cho-Ying and Neumann, Ulrich},
  booktitle={ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={2490--2494},
  year={2021},
  organization={IEEE}
}

Acknowledgement

The code development is based on CFCNet, Self-Supervised Depth Completion, and PSMNet.

scadc-depthcompletion's Issues

[evaluate.py] error : unrecognized data path

Hello, I am a student who is interested in sensor fusion.

To run SCADC-DepthCompletion, I downloaded the data as it was posted on README.md and executed the following code.

However, there was an error that did not recognize the test data path even though the data file was downloaded normally.

So I tried to set up a number of data paths to solve this problem, but I couldn't.

I'd like some advice on how to solve this problem.

By the way, your repositories is very interesting for me, who wants to go to graduate school in the field of computer vision.

I also run CFCNet repository you uploaded, got the same image as the result in the repository and I read the paper to study the model structure.

The reason for running CFCNet and SCADC-DeepthCompletion is to learn more about deep learning models.

As a student who dreams of becoming a self-driving car's vision SW engineer, I am very impressed your repositories.

I hope you have a good day today, and thank you for reading the git issue!