Self-supervised Depth and Ego-motion Estimation for Monocular Thermal Video using Multi-spectral Consistency Loss
This github implements the system described in the paper:
Self-supervised Depth and Ego-motion Estimation for Monocular Thermal Video using Multi-spectral Consistency Loss
Ukcheol Shin, Kyunghyun Lee, Seokju Lee, In So Kweon
Robotics and Automation Letter 2022 & ICRA 2022
[PDF] [Project webpage] [Full paper] [Youtube]
This codebase was developed and tested with python 3.7, Pytorch 1.5.1, and CUDA 10.2 on Ubuntu 16.04.
conda env create --file environment.yml
See "scripts/run_prepare_vivid_data.sh".
For ViViD Raw dataset, download the dataset provided on the official website.
For our post-processed dataset and pre-trained models, you can download after fill out a simple survey.
We will send you an e-mail with a download link.
After download post-processed dataset, generate training/testing dataset
sh scripts/run_prepare_vivid_data.sh
The "scripts" folder provides several examples for training and testing.
You can train the depth and pose model on vivid dataset by running
sh scripts/train_vivid_resnet18_indoor.sh
sh scripts/train_vivid_resnet18_outdoor.sh
Then you can start a tensorboard
session in this folder by
tensorboard --logdir=checkpoints/
and visualize the training progress by opening https://localhost:6006 on your browser.
You can evaluate depth and pose by running
sh scripts/test_vivid_indoor.sh
sh scripts/test_vivid_outdoor.sh
and visualize depth by running
sh scripts/run_inference.sh
Models | Abs Rel | Sq Rel | RMSE | RMSE(log) | Acc.1 | Acc.2 | Acc.3 |
---|---|---|---|---|---|---|---|
Ours(T) | 0.231 | 0.215 | 0.730 | 0.266 | 0.616 | 0.912 | 0.990 |
Ours(MS) | 0.163 | 0.123 | 0.553 | 0.204 | 0.771 | 0.970 | 0.995 |
Models | Abs Rel | Sq Rel | RMSE | RMSE(log) | Acc.1 | Acc.2 | Acc.3 |
---|---|---|---|---|---|---|---|
Ours(T) | 0.157 | 1.179 | 5.802 | 0.211 | 0.750 | 0.948 | 0.985 |
Ours(MS) | 0.146 | 0.873 | 4.697 | 0.184 | 0.801 | 0.973 | 0.993 |
Metric | ATE | RE |
---|---|---|
Ours(T) | 0.0063 | 0.0092 |
Ours(MS) | 0.0057 | 0.0089 |
Metric | ATE | RE |
---|---|---|
Ours(T) | 0.0571 | 0.0280 |
Ours(MS) | 0.0562 | 0.0287 |
Please cite the following paper if you use our work, parts of this code, and pre-processed dataset in your research.
@article{shin2021self,
title={Self-supervised Depth and Ego-motion Estimation for Monocular Thermal Video using Multi-spectral Consistency Loss},
author={Shin, Ukcheol and Lee, Kyunghyun and Lee, Seokju and Kweon, In So},
journal={IEEE Robotics and Automation Letters},
year={2021},
publisher={IEEE}
}
- SfMLearner-Pytorch (CVPR 2017)
- SC-SfMLearner-Pytorch (NeurIPS 2019)