yangcaoai / coda_neurips2023

Official code for NeurIPS2023 paper: CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection

📖 CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection (NeurIPS2023)

🔥Please star CoDA ⭐ and share it. Thanks🔥

[Paper]   [Project Page]

Yang Cao, Yihan Zeng, Hang Xu, Dan Xu
The Hong Kong University of Science and Technology
Huawei Noah's Ark Lab

🚩 Updates

☑ Our extended work CoDAv2 is released, check out it on arXiv !

☑ Latest papers&codes about open-vocabulary perception are collected here.

☑ All the codes, data and pretrained models have been released!

☑ The training and testing codes have been released.

☑ The pretrained models have been released.

☑ The OV-setting SUN-RGBD datasets have been released.

☑ The OV-setting ScanNet datasets have been released.

☑ Paper LaTeX codes are available at https://scienhub.com/Yang/CoDA.

Framework

Samples

Installation

Our code is based on PyTorch 1.8.1, torchvision==0.9.1, CUDA 10.1 and Python 3.7. It may work with other versions.

Please also install the following Python dependencies:

matplotlib
opencv-python
plyfile
'trimesh>=2.35.39,<2.35.40'
'networkx>=2.2,<2.3'
scipy

Please install pointnet2 layers by running

cd third_party/pointnet2 && python setup.py install

Please install a Cythonized implementation of gIOU for faster training.

conda install cython
cd utils && python cython_compile.py build_ext --inplace

Dataset preparation

To achieve the OV setting, we re-organize the original ScanNet and SUN RGB-D and adopt annotations of more categories. Please directly download the ov-setting datasets we provide here: OV SUN RGB-D and OV ScanNet.

Then run for the downloaded *.tar file:

bash data_preparation.sh

Evaluation

Download the pretrained models here. Then run:

bash test_release_models.sh

Training

bash scripts/coda_sunrgbd_stage1.sh
bash scripts/coda_sunrgbd_stage2.sh

Running Samples

bash run_samples.sh

📜 BibTeX

If CoDA is helpful, please cite:

@inproceedings{cao2023coda,
  title={CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection},
  author={Cao, Yang and Zeng, Yihan and Xu, Hang  and  Xu, Dan},
  booktitle={NeurIPS},
  year={2023}
}
@article{cao2024collaborative,
      title={Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detection}, 
      author={Yang Cao and Yihan Zeng and Hang Xu and Dan Xu},
    journal={arXiv preprint arXiv:2406.00830},
    year={2024}
}

📧 Contact

If you have any question or collaboration need (research purpose or commercial purpose), please email [email protected].

📜 Acknowledgement

CoDA is inspired by CLIP and 3DETR. We appreciate their great codes.

coda_neurips2023's Issues

Training Error

Dear Authors,

When I train this code, there is an error.

Training Error

Dear Authors,

When I train this code, there still exists an error:

Code Release

Thank you for your excellent jobs，it is so cool!!! I wonder if you will release the code? It is important for me. Waiting for your reply!

How to organize the OV-setting SUN-RGBD datasets?

I've downloaded the file named sunrgbd_v1_revised_0415.tar, is this file enough for testing and evaluation process?

Training Time

Dear Authors,

How long does your method need to train?

The train errors AttributeError: 'Model3DETRPredictedBoxDistillationHead' object has no attribute 'if_select_box_by_objectness'

I encountered the following problem when running bash scripts/coda_sunrgbd_stage1.sh

When reviewing the code, I found that during initialization, the default setting is if_select_box_by_objectness=False. Do I need to modify the code?

The issues of "python: can't open file 'cloud_tools/mox.py': [Errno 2] No such file or directory"

Could you please explain the purpose of the cloud_tools/mox.py file? Do I need to make any modifications to it?

About Training

Dear Authors,

I have downloaded your given data. But I do not run your code. There exist multiple errors：

Data downloading

Thanks for your contribution for this nice work! I was setting up the repo but fail to download the large data files (~300GB) from Sharepoint. Do you mind uploading it to somewhere else so that the files can be downloaded via command lines. Thanks!

About categories

Hi,
Thanks for your great work! It seems the authors are busy and the code release may be later. I wonder if you can share the names of base and novel categories on ScanNet and SUN-RGBD, that will make others easier to compare with CoDA. Thanks a lot!

File Error

Dear Authors,

When I run this code, there exists an error:

FileNotFoundError: [Errno 2] No such file or directory: 'Data/sunrgb_d/sunrgbd_v1_revised_0415/sunrgbd_pc_bbox_votes_50k_v1_all_classes_revised_0415_minival'

Dataset could not access

https://hkustconnect-my.sharepoint.com/:f:/g/personal/ycaobd_connect_ust_hk/EqeHrtMq5aZBmnZUR79vub4B9HZIBxYLd5K7z6VuvemHzg?e=e7ooDv
can not open now.