This repo implements the system described in the CVPR-2018 paper:
Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, Ian Reid
@InProceedings{Zhan_2018_CVPR,
author = {Zhan, Huangying and Garg, Ravi and Saroj Weerasekera, Chamara and Li, Kejie and Agarwal, Harsh and Reid, Ian},
title = {Unsupervised Learning of Monocular Depth Estimation and Visual Odometry With Deep Feature Reconstruction},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2018}
}
This repo includes (1) the training procedure of our models; (2) evaluation scripts for the results; (3) trained models and results.
- Requirements
- Prepare dataset
- Depth
- Depth and odometry
- Feature Reconstruction Loss for Depth
- Depth, odometry and feature
- Result evaluation
This code was tested with Python 2.7, CUDA 8.0 and Ubuntu 14.04 using Caffe.
Caffe: Add the required layers in ./caffe into your own Caffe. Remember to enable Python Layers in the Caffe configuration.
Most of our required models, trained models and results can be downloaded from here. The following instruction also includes specific links to the items.
The main dataset used in this project is KITTI Driving Dataset. Please follow the instruction in ./data/README.md to prepare the required dataset.
For our trained models and pre-requested models, please visit here to download the models and put the models into the directory ./models.
In this part, the training of single view depth estimation network from stereo pairs is introduced. Photometric loss is used as the main supervision signal. Only stereo pairs are used in this experiment.
- Update
$YOUR_CAFFE_DIRin./experiments/depth/train.sh. - Run
bash ./expriments/depth/train.sh.
The trained models are saved in ./snapshots/depth
In this part, the joint training of the depth estimation network and the visual odometry network is introduced. Photometric losses for spatial pairs and temporal pairs are used as the main supervision signal. Both spatial (stereo) pairs and temporal pairs (i.e. stereo sequences) are used in this experiment.
To facilitate the training, the model trained in the Depth experiment is used as an initialization.
- Update
$YOUR_CAFFE_DIRin./experiments/depth_odometry/train.sh. - Run
bash ./expriments/depth_odometry/train.sh.
The trained models are saved in ./snapshots/depth_odometry
In this part, the training of single view depth estimation network from stereo pairs is introduced. Both photometric loss and feature reconstruction loss are used as the main supervision signal. Only stereo pairs are used in this experiment. There are several features we have tried for this experiment. Currently, only the example of using KITTI Feat. is shown here. More details of using other features will be updated later.
To facilitate the training, the model trained in the Depth experiment is used as an initialization.
- Update
$YOUR_CAFFE_DIRin./experiments/depth_feature/train.sh. - Run
bash ./expriments/depth_feature/train.sh.
The trained models are saved in ./snapshots/depth_feature
In this part, we show the training including feature reconstruction loss. Stereo sequences are used in this experiment.
With the feature extractor proposed in Weerasekera et.al, we can finetune the trained depth model and/or odometry model with our proposed deep feature reconstruction loss.
- Update
$YOUR_CAFFE_DIRin./experiments/depth_odometry_feature/train.sh. - Run
bash ./expriments/depth_odometry_feature/train.sh.
NOTE: The link to download the feature extractor proposed in Weerasekera et.al will be released soon.
Note that the evaluation script provided here uses a different image interpolation for resizing input images (i.e. python's interpolation v.s. Caffe's interpolation), therefore the quantative result could be a little different from the published result.
Using the test set (697 image-depth pairs from 28 scenes) in Eigen Split is a common protocol to evaluate depth estimation result.
We basically use the evaluation script provided by monodepth to evalute depth estimation results.
In order to run the evaluation, a npy file is required to store the predicted depths. Then run the script to evaluate the performance.
- Update
caffe_rootin./tools/evaluation_tools.py - To generate the depth prediction and save it in a
npyfile.
python ./tools/evaluation_tools.py --func generate_depth_npy --dataset kitti_eigen --depth_net_def ./experiments/networks/depth_deploy.prototxt --model models/trained_models/eigen_split/Baseline.caffemodel --npy_dir ./result/depth/inv_depths_baseline.npy
- To evalute the predictions.
python ./tools/eval_depth.py --split eigen --predicted_inv_depth_path ./result/depth/inv_depths_baseline.npy --gt_path data/kitti_raw_data/ --min_depth 1 --max_depth 50 --garg_crop
Some of our results (inverse depths) are released and can be downloaded from here.
KITTI Odometry benchmark contains 22 stereo sequences, in which 11 sequences are provided with ground truth. The 11 sequences are used for evaluation or training of visual odometry.
- Update
caffe_rootin./tools/evaluation_tools.py - To generate the odometry predictions (relative camera motions), run the following script.
python ./tools/evaluation_tools.py --func generate_odom_result --model models/trained_models/odometry_split/Temporal.caffemodel --odom_net_def ./experiments/networks/odometry_deploy.prototxt --odom_result_dir ./result/odom_result
- After getting the odometry predictions, we can evalute the performance by comparing with the ground truth poses.
python ./tools/evaluation_tools.py --func eval_odom --odom_result_dir ./result/odom_result
Our odometry results are released and can be downloaded from here.
For academic usage, the code is released under the permissive BSD license. For any commercial purpose, please contact the authors.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent