antabangun / coex

Hello there! 👋

I'm Antyanta Bangunharcana, a PhD candidate at the Mechatronics, Systems, and Control Lab in the Department of Mechanical Engineering at KAIST. My research focuses on 3D computer vision, with a particular interest in depth estimation using cameras and visual localization.

Projects

I have several projects, but two of my most recent ones that I have made public are:

• CoEx: A real-time stereo matching algorithm that achieves state-of-the-art performance on various benchmark datasets. This work was published in IROS 2021.
• DualRefine: A self-supervised depth and pose estimation algorithm that uses iterative epipolar sampling and refinement to achieve high accuracy. This work was presented at CVPR 2023.

Feel free to check out the repositories and let me know if you have any questions or comments.

Contact

You can reach me at [email protected]. You can also find me on LinkedIn and Google Scholar.

coex's Issues

torch.argsort

Thanks again for this work. This issue is more-so to document our initial conversation attempts to get this to run with DepthAI. The first step is to convert to ONNX, and then from there OpenVINO is used to make it run on the Myriad X.

Katsuya Hyodo (PINTO0309) is working on the conversion to ONNX now, and ran into torch.argsort not being supported in ONNX, so he’s making an attempt at working around it now.

Full ONNX conversion error message below:

Traceback (most recent call last):
  File "demo.py", line 49, in <module>
    torch.onnx.export(pose_ssstereo, (x, x), f"coex_{H}x{W}.onnx", opset_version=13)
  File "/usr/local/lib/python3.8/dist-packages/torch/onnx/__init__.py", line 275, in export
    return utils.export(model, args, f, export_params, verbose, training,
  File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 88, in export
    _export(model, args, f, export_params, verbose, training, input_names, output_names,
  File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 689, in _export
    _model_to_graph(model, args, verbose, input_names,
  File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 463, in _model_to_graph
    graph = _optimize_graph(graph, operator_export_type,
  File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 200, in _optimize_graph
    graph = torch._C._jit_pass_onnx(graph, operator_export_type)
  File "/usr/local/lib/python3.8/dist-packages/torch/onnx/__init__.py", line 313, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 990, in _run_symbolic_function
    symbolic_fn = _find_symbolic_in_registry(domain, op_name, opset_version, operator_export_type)
  File "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py", line 944, in _find_symbolic_in_registry
    return sym_registry.get_registered_op(op_name, domain, opset_version)
  File "/usr/local/lib/python3.8/dist-packages/torch/onnx/symbolic_registry.py", line 116, in get_registered_op
    raise RuntimeError(msg)
RuntimeError: Exporting the operator argsort to ONNX opset version 13 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.

Problem of metric in SceneFlow dataset

Hello, thanks for your good job.
I want to re-train the models with SceneFlow dataset. In the file "cfg_coex.yaml", I set "sceneflow" and "kitti15" are ture. I'm sure I don't change the code. But the test metric is bad.

I don't know what the problem is, I look forward to your reply.

Question about Application - visual odometry and point cloud mapping

Outstanding work, may I ask how the combination of depth and VO system demonstrated on your project website was done, can you describe the process?

cuda initialization failed

Hello, there is a problem when I run demo.py

File "demo.py", line 45, in
cfg=cfg).cuda()
RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu.

Is there any solution?

Sweet!

Hi @antabangun ,

This is less of an issue and more of a "Wow, this work looks awesome!".

So we're going to try to get this running on DepthAI. As it would be super cool to have something of this capability running natively on a little embedded camera. And as a TLDR we make an ecosystem of open source hardware, software, and firmware with the end goal being to make it as easy as possible to embed performant spatial AI and CV into little things. https://github.com/luxonis

We're just initially looking at it now, so we don't know the complexity of porting this. But we'll be giving it a shot.

Thanks,
Brandon

Depth map quality

What do I need to do to make it more accurate?

Can't reproduce the result:epe 0.69

Hi! Thank you for the great works!
I'm trying to Re-train the model, But I got epe 0.799957.
I wonder if I ignore some setting.
I only made a few changes in cfg_coex.yaml:

device: [0,1,2,3,4,5,6,7]
train_on:
sceneflow: True
kitti12: False
kitti15: False
kittiraw: False
kitti360: False

Experiments were on 8 3090Ti GPUs
I want to know on which dataset did you train and get epe 0.69?

SceneFlow checkpoints

We would like to get the pretrained model on sceneflow for our fine work, could you please offer it on google? Thanks very much!

About the speed

Hello, thanks for your nice job.
I tried to run coex on my own RTX2080 Ti, however, for kitti data(reso of 376*1241), it takes about 45ms per image(about 22 fps), and I believe there is nobody else using this GPU when I am running the code, do you know why？
one more question, I found that there are usually some artifacts in textureless region such as the red box region of the follow image

.

TensorRT model

Hello @antabangun,
Thank you for the great work!
Have you tried converting the torchscript model to tensorrt?
If so, can you please share either the model or how you did it?
I tried it with torch2trt but it fails since some of the layers are not supported.
Thank you!

Unable to get the result

Hi, @antabangun great work,
I am tring to run demo.py for inference but i am unable to get the output (result).
Demo.py is executed without any error but unable to get the output.
Thankyou

train file

Thank you for your work! I'm a SLAM beginner and want to try using your work in a project, but I know very little about deep learning. Could you please provide the train file? I will be extremely grateful!

KITTI and SceneFlow pretrained model

Hi,
Thanks for your great work. I'm trying to run your code. And I found that you gave a model(logs/stereo/CoEx/version_0/checkpoints/last.ckpt), but I can't reproduce the results of the paper when I tested this model. Is this model from sceneflow or kitti, and whether the pre-training models of both datasets can be opened?

Thank you very much!

test

Hello, sorry to bother you again. Your algorithm impressed me deeply. I try to understand and learn deeply. Please allow me to ask you some questions. How to test my own data set? Does the test require camera calibration parameters similar to those in kittiraw?

kitti checkpoint

Hi,
Thanks for your great work. I'm trying to run your code. I find it, sorry. but I find the checkpoint here is sceneflow-epoch=9.ckpt. Can you also provide the kitt2015 submission checkpoint? Thanks.

The pretrained model error

hello sir, I got the error like this :
If any possibility that the pretrained model you gave was @wrong?

Traceback (most recent call last):
File "/media/hy/DISK2/depth_estimate/6-Coex/demo.py", line 41, in
pose_ssstereo = Stereo.load_from_checkpoint(cfg['stereo_ckpt'],
File "/home/hy/anaconda3/envs/leastereo/lib/python3.8/site-packages/pytorch_lightning/core/saving.py", line 161, in load_from_checkpoint
model = cls._load_model_state(checkpoint, strict=strict, kwargs)
File "/home/hy/anaconda3/envs/leastereo/lib/python3.8/site-packages/pytorch_lightning/core/saving.py", line 203, in _load_model_state
model = cls(_cls_kwargs)
File "/media/hy/DISK2/depth_estimate/6-Coex/stereo.py", line 44, in init
self.stereo = load_class(
File "/media/hy/DISK2/depth_estimate/6-Coex/models/stereo/CoEx.py", line 26, in init
self.feature = Feature(self.cfg)
File "/media/hy/DISK2/depth_estimate/6-Coex/models/stereo/submodules/feature.py", line 204, in init
model = timm.create_model(self.type, pretrained=pretrained, features_only=True)
File "/home/hy/anaconda3/envs/leastereo/lib/python3.8/site-packages/timm/models/factory.py", line 71, in create_model
model = create_fn(pretrained=pretrained, pretrained_cfg=pretrained_cfg, **kwargs)
File "/home/hy/anaconda3/envs/leastereo/lib/python3.8/site-packages/timm/models/efficientnet.py", line 1335, in mobilenetv2_100
model = _gen_mobilenet_v2('mobilenetv2_100', 1.0, pretrained=pretrained, **kwargs)
File "/home/hy/anaconda3/envs/leastereo/lib/python3.8/site-packages/timm/models/efficientnet.py", line 766, in _gen_mobilenet_v2
model = _create_effnet(variant, pretrained, **model_kwargs)
File "/home/hy/anaconda3/envs/leastereo/lib/python3.8/site-packages/timm/models/efficientnet.py", line 629, in _create_effnet
model = build_model_with_cfg(
File "/home/hy/anaconda3/envs/leastereo/lib/python3.8/site-packages/timm/models/helpers.py", line 545, in build_model_with_cfg
load_pretrained(
File "/home/hy/anaconda3/envs/leastereo/lib/python3.8/site-packages/timm/models/helpers.py", line 245, in load_pretrained
state_dict = load_state_dict_from_url(
File "/home/hy/anaconda3/envs/leastereo/lib/python3.8/site-packages/torch/hub.py", line 595, in load_state_dict_from_url
return torch.load(cached_file, map_location=map_location)
File "/home/hy/anaconda3/envs/leastereo/lib/python3.8/site-packages/torch/serialization.py", line 713, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/hy/anaconda3/envs/leastereo/lib/python3.8/site-packages/torch/serialization.py", line 938, in _legacy_load
typed_storage._storage._set_from_file(
RuntimeError: unexpected EOF, expected 139955 more bytes. The file might be corrupted.

cuda out of memory in training

Hello, first I will say this is a great work@antabangun
This issue is about training. For example, When cfg['training']['sceneflow_max_epochs'] = 10, the training starts from epoch=0 to epoch=10. When last epoch starts(just epoch=10), the cuda memory increase largely(just from 10.5GB to 16.5GB), and the training time becomes quickly!(from 18min/epoch to 7min/epoch)

So, when I trained on GPU(memory=11GB), I will appear this mistake: