hustvl / osp

[ECCV 2024] Occupancy as Set of Points

Occupancy as Set of Points

Yiang Shi^1,*, Tianheng Cheng^1,*, Qian Zhang², Wenyu Liu¹, Xinggang Wang^{1 📧}

¹ School of EIC, HUST, ² Horizon Robotics

* equal contribution, ^📧 corresponding author.

ECCV 2024

News

• [2024-7-8] We have released the arXiv paper of OSP.
• [2024-7-2] OSP is accepted by ECCV 2024!

Abstract

In this paper, we explore a novel point representation for 3D occupancy prediction from multi-view images, which is named Occupancy as Set of Points. Existing camera-based methods tend to exploit dense volume-based representation to predict the occupancy of the whole scene, making it hard to focus on the special areas or areas out of the perception range. In comparison, we present the \textit{Points of Interest} (PoIs) to represent the scene and propose OSP, a novel framework for point-based 3D occupancy prediction. Owing to the inherent flexibility of the point-based representation, OSP achieves strong performance compared with existing methods and excels in terms of training and inference adaptability. It extends beyond traditional perception boundaries and can be seamlessly integrated with volume-based methods to significantly enhance their effectiveness. Experiments on the Occ3D-nuScenes occupancy benchmark show that OSP has strong performance and flexibility.

Preliminary

Installation

1. Prepare conda environment referring to the documentation of BEVFormer

Prepare Dataset

1. Download nuScenes and prepare annotations referring to the documentation of 3D Occupancy Prediction Challenge at CVPR 2023

Pretrained Weights

The pretrained weight of fcos3d can be downloaded here

Usage

1. Training

   bash train.sh

   • Replace the default config file as needed.
   • Config osp_minibatch.py represents mini dataset of nuScenes.

2. Evaluation

   bash test.sh

   • Replace the default config file as needed.
   • Replace the checkpoint path in the script with your own.

Results

Backbone	Method	Lr Schd	IoU	Config	Download
R101	OSP	24ep	39.41	config	model
R101	BEVFormer w/ OSP	24ep	41.21	config	model

• Model weights will be released later.

Citations

@inproceedings{shi2024occupancysetpoints,
      title={Occupancy as Set of Points}, 
      author={Yiang Shi and Tianheng Cheng and Qian Zhang and Wenyu Liu and Xinggang Wang},
      year={2024},
      booktitle={ECCV}
}

License

Released under the MIT License.

osp's Issues

'mask_camera' question

How to get 'mask_camera' when you run model inference in vehicle?

Config and code modification for BEVFormer w/OSP

Hi, Thanks for your good job!
I am trying to reproduce the result(mIoU=41.21) of BEVFormer w/OSP, but can not find the correct config. In your README-Results, OSP and BEVFormer w/OSP share the same config(projects/configs/osp.py), could you please provide the detailed config for BEVFormer w/OSP(e.g., the config to freeze trained BEVFormer)? Besides, I have some other questions:

1. why self-attn osp.py is not needed in OSP?
2. there are 3 encoder layers, but each layer forward twice BEVFormerEncoder, why? Any explanation is appreciated.
3. i find a hard-code use_bev_volume=False, is this should be True for using OSP as a plugin to argument volume-based methods? and how to modify the code?

Looking forward to your reply, Thanks!