nburrus / stereodemo

Small Python utility to compare and visualize the output of various stereo depth estimation algorithms

stereodemo

Small Python utility to compare and visualize the output of various stereo depth estimation algorithms:

• Make it easy to get a qualitative evaluation of several state-of-the-art models in the wild
• Feed it left/right images or capture live from an OAK-D camera
• Interactive colored point-cloud view since nice-looking disparity images can be misleading
• Try different parameters on the same image

Included methods (implementation/pre-trained models taken from their respective authors):

• OpenCV stereo block matching and Semi-global block matching baselines, with all their parameters

• CREStereo: "Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation" (CVPR 2022)

• RAFT-Stereo: "Multilevel Recurrent Field Transforms for Stereo Matching" (3DV 2021)

• Hitnet: "Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching" (CVPR 2021)

• STereo TRansformers: "Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers" (ICCV 2021)

• Chang et al. RealtimeStereo: "Attention-Aware Feature Aggregation for Real-time Stereo Matching on Edge Devices" (ACCV 2020)

• DistDepth: "Toward Practical Monocular Indoor Depth Estimation" (CVPR 2022). This one is actually a monocular method, only using the left image.

See below for more details / credits to get each of these working, and check this blog post for more results, including performance numbers.

stereodemo-intro.mp4

Getting started

Installation

python3 -m pip install stereodemo

Running it

With an OAK-D camera

To capture data directly from an OAK-D camera, use:

stereodemo --oak

Then click on Next Image to capture a new one.

With image files

If you installed stereodemo from pip, then just launch stereodemo and it will show some embedded sample images captured with an OAK-D camera.

A tiny subset of some popular datasets is also included in this repository. Just provide a folder to stereodemo and it'll look for left/right pairs (either im0/im1 or left/right in the names):

# To evaluate on the oak-d images
stereodemo datasets/oak-d 

# To cycle through all images
stereodemo datasets

Then click on Next Image to cycle through the images.

Sample images included in this repository:

• drivingstereo: outdoor driving.
• middlebury_2014: high-res objects.
• eth3d: outdoor and indoor scenes.
• sceneflow: synthetic rendering of objects.
• oak-d: indoor images I captured with my OAK-D lite camera.
• kitti2015: outdoor driving (only one image).

Dependencies

pip will install the dependencies automatically. Here is the list:

• Open3D. For the point cloud visualization and the GUI.
• OpenCV. For image loading and the traditional block matching baselines.
• onnxruntime. To run pretrained models in the ONNX format.
• pytorch. To run pretrained models exported as torch script.
• depthai. Optional, to grab images from a Luxonis OAK camera.

Credits for each method

I did not implement any of these myself, but just collected pre-trained models or converted them to torch script / ONNX.

• CREStereo

  • Official implementation and pre-trained models: https://github.com/megvii-research/CREStereo
  • Model Zoo for the ONNX models: https://github.com/PINTO0309/PINTO_model_zoo/tree/main/284_CREStereo
  • Port to ONNX + sample loading code: https://github.com/ibaiGorordo/ONNX-CREStereo-Depth-Estimation

• RAFT-Stereo

  • Official implementation and pre-trained models: https://github.com/princeton-vl/RAFT-Stereo
  • I exported the pytorch implementation to torch script via tracing, with minor modifications of the source code.
  • Their fastest implementation was not imported.

• Hitnet

  • Official implementation and pre-trained models: https://github.com/google-research/google-research/tree/master/hitnet
  • Model Zoo for the ONNX models: https://github.com/PINTO0309/PINTO_model_zoo/tree/main/142_HITNET
  • Port to ONNX + sample loading code: https://github.com/ibaiGorordo/ONNX-HITNET-Stereo-Depth-estimation

• Stereo Transformers

  • Official implementation and pre-trained models: https://github.com/mli0603/stereo-transformer
  • Made some small changes to allow torch script export via tracing.
  • The exported model currently fails with GPU inference, so only CPU inference is enabled.

• Chang et al. RealtimeStereo

  • Official implementation and pre-trained models: https://github.com/JiaRenChang/RealtimeStereo
  • I exported the pytorch implementation to torch script via tracing with some minor changes to the code JiaRenChang/RealtimeStereo#15 . See chang_realtimestereo_to_torchscript_onnx.py.

• DistDepth

  • Official implementation and pre-trained models https://github.com/facebookresearch/DistDepth
  • I exported the pytorch implementaton to torch script via tracing, see the changes.

License

The code of stereodemo is MIT licensed, but the pre-trained models are subject to the license of their respective implementation.

The sample images have the license of their respective source, except for datasets/oak-d which is licenced under Creative Commons Attribution 4.0 International License.

stereodemo's Issues

Correct version of open3d

Upon first installation, it would install open3d v0.10.0.0, but it need the latest version v0.15.2

First error after installation is the following:
import open3d.visualization.gui as gui
After manually installing v0.12.0 as recommended here, I get the following error:
AttributeError: 'open3d.cpu.pybind.visualization.gui.SceneWidget' object has no attribute 'look_at'
which is new in v0.13.0, so I go ahead and install that, and I get the following error:
AttributeError: 'open3d.cpu.pybind.visualization.gui.ListView' object has no attribute 'set_max_visible_items'
and well then I installed v0.15.2, and that finally stopped complaining.
Might be best to add more strict requirements to the setup config.

Though it still doesn't start well, window doesn't render and nothing happens. Does it download models in the background? It should probably have some indication of that before opening a GUI window. Output:

INFO: will store downloaded models in /home/user/.cache/stereodemo/models
Trying to connect to an OAK camera...
FEngine (64 bits) created at 0x7ff6390 (threading is enabled)
FEngine resolved backend: OpenGL

Stereo Algorithms (Include:CREStereo,RAFT-Stereo,Hitnet,FastACVNet_plus,Stereo Transformers,RealtimeStereo,DistDepth) with TensorRT,ORT,OpenVINO

https://github.com/pcb9382/StereoAlgorithms

The Stereo algorithm is implemented using TensorRT

https://github.com/pcb9382/StereoAlgorithms

OV9282 doesn't support THE_480_P resolution, defaulting to THE_400_P

Thanks for the wonderful code.
I tried running the demo using OAK-D, was getting error based on resolution.

[14442C1061F35DD700] [2.4] [510.014] [MonoCamera(1)] [error] OV9282 doesn't support THE_480_P resolution, defaulting to THE_400_P
[14442C1061F35DD700] [2.4] [510.014] [MonoCamera(0)] [error] OV9282 doesn't support THE_480_P resolution, defaulting to THE_400_P

assert self.input.left_image.shape[1] == self.input.calibration.width and self.input.left_image.shape[0] == self.input.calibration.height

The demo without the OAK-D was working.

Strange results

I use my own data without calibration data, and got this map.Does the calibration is always required?

GUI window is closed

I have an issue in that the GUI window is closed.
Can you solve it?