owen-liuyuxuan / fsnet

Full Scale Monocular Depth Prediction. Official Implementation of "FSNet: Redesign Self-Supervised MonoDepth for Full-Scale Depth Prediction for Autonomous Driving" https://arxiv.org/abs/2304.10719

FSNet

This repo aims to provide a flexible and reproducible Full-Scale Unsupervised Monocular Depth Prediction from well-calibrated/tracked image sequences.

The repo share a similar design strategy with visualDet3D, but we adopt a even more flexible design for data processing and config construction, allowing easy insertion of new tasks/models while not interfere with the existing ones.

We provide cookbooks and solutions for Full-Scale Unsupervised Depth Prediction in KITTI, KITTI360, NuScenes, and KITTI-360 Fisheye. Links are directed to their corresponding cookbooks. We hope we could make training/testing/deploying/ROS demo easier.

Reference: this repo borrow codes and ideas from visualDet3D, monodepth2.

Setup:

Environment setup

pip3 install -r requirement.txt

or manually check dependencies.

Docker Setup

cd docker
docker build ./ -t fsnet

The image will contain necessary packages for the repo.

Training / Testing

Please check the corresponding datasets: KITTI, KITTI360, NuScenes, and KITTI-360 Fisheye. More will be available through contributions and further paper submission.

Config and Path setup

Please modify the path and other parameters in config/*.py. config/*_example files are templates.

Notice: *_examples are NOT utilized by the code and *.py under /config is ignored by .gitignore.

The content of the selected config file will be recorded in tensorboard at the beginning of training.

important paths to modify in config :

1. cfg.path.{data}_path: Path to the data directory.
2. cfg.path.project_path: Path to the workdirs of the projects (will have temp_outputs, log, checkpoints)

Multi-Dataset Training

We can use the ConcatDataset APIs to jointly train one depth prediction network using multiple datasets. The output of each dataset indexing methods should output dictionaries with same keys and same tensor shapes, but we could define different data-proprocessing pipelines for each dataset and align multiple datasets if needed.

Build Module Path

In the project, most of the modules are initialized by builder and find_object. The builder will import and initialize the target object/function based on the full import path. Please check the example configs.

Further Info and Bug Issues

1. Open issues on the repo if you meet troubles or find a bug or have some suggestions.
2. Email to [email protected]

Other Resources

• RAM-LAB
• Collections of Papers and Readings;

fsnet's Issues

KITTI360 fisheye experiment

Hello,

I would like to express my appreciation for your impressive work.

I am currently attempting to replicate the results of the KITTI360 fisheye experiment as described in your paper and provided in the repository. However, I am encountering some difficulties in achieving the reported results.

1. I trained the model using the original code and settings for 20 epochs. The results I obtained were different from those reported in the paper.

2. To address this, I increased the batch size and extended the training to 50 epochs. Even with these adjustments, the results remain not good.

Thank you for your time and assistance, again.

How to use flow mask?

Hello, in the paper, it was mentioned that using optical flow to calculate the mask of moving objects.

But in all configs of this repo, motion_ mask、is_precompute_flow are all False, is this feature not yet open?

weights and training preparation

Can you plan to release pre-trained weights and add installation readme and dataset preparation readme?

Nuscenes dataset problems

Thank you for your sharing!
When I trained on Nuscenes Dataset , I followed every step in readme doc.
But I found FSNET only have 1/3 output pics compared with Nuscenes-full dataset ,also there was no pics in CAM_BACK view.
Did I do something wrong?

this is the result of nuscenes training set

this is the result of nuscenes val set