tianweiy / mvp

Multimodal Virtual Point 3D Detection

Turning pixels into virtual points for multimodal 3D object detection.

Multimodal Virtual Point 3D Detection,
Tianwei Yin, Xingyi Zhou, Philipp Krähenbühl,
arXiv technical report (arXiv 2111.06881 )

@article{yin2021multimodal,
  title={Multimodal Virtual Point 3D Detection},
  author={Yin, Tianwei and Zhou, Xingyi and Kr{\"a}henb{\"u}hl, Philipp},
  journal={NeurIPS},
  year={2021},
}

Contact

Any questions or suggestions are welcome!

Tianwei Yin [email protected] Xingyi Zhou [email protected]

Abstract

Lidar-based sensing drives current autonomous vehicles. Despite rapid progress, current Lidar sensors still lag two decades behind traditional color cameras in terms of resolution and cost. For autonomous driving, this means that large objects close to the sensors are easily visible, but far-away or small objects comprise only one measurement or two. This is an issue, especially when these objects turn out to be driving hazards. On the other hand, these same objects are clearly visible in onboard RGB sensors. In this work, we present an approach to seamlessly fuse RGB sensors into Lidar-based 3D recognition. Our approach takes a set of 2D detections to generate dense 3D virtual points to augment an otherwise sparse 3D point-cloud. These virtual points naturally integrate into any standard Lidar-based 3D detectors along with regular Lidar measurements. The resulting multi-modal detector is simple and effective. Experimental results on the large-scale nuScenes dataset show that our framework improves a strong CenterPoint baseline by a significant 6.6 mAP, and outperforms competing fusion approaches.

Main results

3D detection on nuScenes validation set

	MAP ↑	NDS ↑
CenterPoint-Voxel	59.5	66.7
CenterPoint-Voxel + MVP	66.0	69.9
CenterPoint-Pillar	52.4	61.5
CenterPoint-Pillar + MVP	62.8	66.2

3D detection on nuScenes test set

	MAP ↑	NDS ↑	PKL ↓
MVP	66.4	70.5	0.603

Use MVP

Installation

Please install CenterPoint and CenterNet2. Make sure to add a link to CenterNet2 folder in your python path. We will use CenterNet2 for 2D instance segmentation and CenterPoint for 3D detection.

Getting Started

Download nuscenes data and organise as follows

# For nuScenes Dataset         
└── NUSCENES_DATASET_ROOT
       ├── samples       <-- key frames
       ├── sweeps        <-- frames without annotation
       ├── maps          <-- unused
       ├── v1.0-trainval <-- metadata

Create a symlink to the dataset root in both CenterPoint and MVP's root directories.

mkdir data && cd data
ln -s DATA_ROOT nuScenes

Remember to change the DATA_ROOT to the actual path in your system.

Generate Virtual Points

You can download the precomputed virtual points from here.

If you prefer to generating the virtual points yourself. Download the centernet2 model from here and place it in the root directory.

Use the following command in the current directory to generate virtual points for nuscenes training and validation sets. The points will be saved to data/nuScenes/samples or sweeps/LIDAR_TOP_VIRTUAL.

python virtual_gen.py --info_path data/nuScenes/infos_train_10sweeps_withvelo_filter_True.pkl  MODEL.WEIGHTS centernet2_checkpoint.pth 

You will need about 80GB space and the whole process will take 10 to 20 hours using a single GPU.

Create Data

Go to the CenterPoint's root directory and run

# nuScenes
python tools/create_data.py nuscenes_data_prep --root_path=NUSCENES_TRAINVAL_DATASET_ROOT --version="v1.0-trainval" --nsweeps=10 --virtual True 

To save time and space, before running the above command, you can also remove lines below https://github.com/tianweiy/CenterPoint/blob/master/tools/create_data.py#L13 to avoid generating gt database. After that, remember to set https://github.com/tianweiy/CenterPoint/blob/master/configs/mvp/nusc_centerpoint_pp_fix_bn_z_scale_virtual.py#L135 to None. The improvements of gt sampling on nuscenes is marginal (<0.5nds).

if you want to reproduce CenterPoint baseline's results, then also run the following command

# nuScenes
python tools/create_data.py nuscenes_data_prep --root_path=NUSCENES_TRAINVAL_DATASET_ROOT --version="v1.0-trainval" --nsweeps=10 --virtual False 

In the end, the data and info files should be organized as follows

# For nuScenes Dataset 
└── CenterPoint
       └── data    
              └── nuScenes 
                     ├── maps          <-- unused
                     |── v1.0-trainval <-- metadata and annotations
                     |── infos_train_10sweeps_withvelo_filter_True.pkl <-- train annotations
                     |── infos_val_10sweeps_withvelo_filter_True.pkl <-- val annotations
                     |── dbinfos_train_10sweeps_withvelo_virtual.pkl <-- GT database info files
                     |── gt_database_10sweeps_withvelo_virtual <-- GT database 
                     |── samples       <-- key frames
                        |── LIDAR_TOP
                        |── LIDAR_TOP_VIRTUAL
                     └── sweeps       <-- frames without annotation
                        |── LIDAR_TOP
                        |── LIDAR_TOP_VIRTUAL

Train & Evaluate in Command Line

Go to CenterPoint's root directory and use the following command to start a distributed training using 4 GPUs. The models and logs will be saved to work_dirs/CONFIG_NAME

python -m torch.distributed.launch --nproc_per_node=4 ./tools/train.py CONFIG_PATH

For distributed testing with 4 gpus,

python -m torch.distributed.launch --nproc_per_node=4 ./tools/dist_test.py CONFIG_PATH --work_dir work_dirs/CONFIG_NAME --checkpoint work_dirs/CONFIG_NAME/latest.pth 

For testing with one gpu and see the inference time,

python ./tools/dist_test.py CONFIG_PATH --work_dir work_dirs/CONFIG_NAME --checkpoint work_dirs/CONFIG_NAME/latest.pth --speed_test 

MODEL ZOO

We experiment with VoxelNet and PointPillars architectures on nuScenes.

VoxelNet

Model	Validation MAP	Validation NDS	Link
centerpoint_baseline	59.5	66.7	URL
Ours	66.0	69.9	URL
Ours + Two Stage	67.0	70.7	URL

PointPillars

Model	Validation MAP	Validation NDS	Link
centerpoint_baseline	52.4	61.5	URL
Ours	62.8	66.2	URL

License

MIT License.

mvp's Issues

Could you please realse a more detailed description to repeat the MVP results in NUS dataset

these keys have mismatched shape:
+-------------------------------+---------------------------------+---------------------------------+
| key | expected shape | loaded shape |
+-------------------------------+---------------------------------+---------------------------------+
| backbone.conv_input.0.weight | torch.Size([16, 3, 3, 3, 5]) | torch.Size([3, 3, 3, 5, 16]) |
| backbone.conv1.0.conv1.weight | torch.Size([16, 3, 3, 3, 16]) | torch.Size([3, 3, 3, 16, 16]) |
| backbone.conv1.0.conv2.weight | torch.Size([16, 3, 3, 3, 16]) | torch.Size([3, 3, 3, 16, 16]) |
| backbone.conv1.1.conv1.weight | torch.Size([16, 3, 3, 3, 16]) | torch.Size([3, 3, 3, 16, 16]) |
| backbone.conv1.1.conv2.weight | torch.Size([16, 3, 3, 3, 16]) | torch.Size([3, 3, 3, 16, 16]) |
| backbone.conv2.0.weight | torch.Size([32, 3, 3, 3, 16]) | torch.Size([3, 3, 3, 16, 32]) |
| backbone.conv2.3.conv1.weight | torch.Size([32, 3, 3, 3, 32]) | torch.Size([3, 3, 3, 32, 32]) |
| backbone.conv2.3.conv2.weight | torch.Size([32, 3, 3, 3, 32]) | torch.Size([3, 3, 3, 32, 32]) |
| backbone.conv2.4.conv1.weight | torch.Size([32, 3, 3, 3, 32]) | torch.Size([3, 3, 3, 32, 32]) |
| backbone.conv2.4.conv2.weight | torch.Size([32, 3, 3, 3, 32]) | torch.Size([3, 3, 3, 32, 32]) |
| backbone.conv3.0.weight | torch.Size([64, 3, 3, 3, 32]) | torch.Size([3, 3, 3, 32, 64]) |
| backbone.conv3.3.conv1.weight | torch.Size([64, 3, 3, 3, 64]) | torch.Size([3, 3, 3, 64, 64]) |
| backbone.conv3.3.conv2.weight | torch.Size([64, 3, 3, 3, 64]) | torch.Size([3, 3, 3, 64, 64]) |
| backbone.conv3.4.conv1.weight | torch.Size([64, 3, 3, 3, 64]) | torch.Size([3, 3, 3, 64, 64]) |
| backbone.conv3.4.conv2.weight | torch.Size([64, 3, 3, 3, 64]) | torch.Size([3, 3, 3, 64, 64]) |
| backbone.conv4.0.weight | torch.Size([128, 3, 3, 3, 64]) | torch.Size([3, 3, 3, 64, 128]) |
| backbone.conv4.3.conv1.weight | torch.Size([128, 3, 3, 3, 128]) | torch.Size([3, 3, 3, 128, 128]) |
| backbone.conv4.3.conv2.weight | torch.Size([128, 3, 3, 3, 128]) | torch.Size([3, 3, 3, 128, 128]) |
| backbone.conv4.4.conv1.weight | torch.Size([128, 3, 3, 3, 128]) | torch.Size([3, 3, 3, 128, 128]) |
| backbone.conv4.4.conv2.weight | torch.Size([128, 3, 3, 3, 128]) | torch.Size([3, 3, 3, 128, 128]) |
| backbone.extra_conv.0.weight | torch.Size([128, 3, 1, 1, 128]) | torch.Size([3, 1, 1, 128, 128]) |
+-------------------------------+---------------------------------+---------------------------------+

Rotation TTA

Could you please share how to do Rotation TTA , eg. [+-6.25°, +-12.5°]
I cannot get the correct results for many many tests.

Visualization

thanks for great work. Ihave some questions:

1- How can I visualize real points and virtual points in BEV (Bird's Eye View) both with and without an image, similar to Figure 1 and Figure 3c and d? Could you please explain how du the figures visualized?

2- Could you please explain how Figure 4 was visualized?

3-and one more question please, in the paper:

For instance segmentation, we use CenterNet2 [73] which adds a cascade RoI heads [3] on top of the first stage proposal network. The overall network runs at 40 FPS and achieves 43.3 instance segmentation mAP on the nuScenes image dataset [2].

What hyperparameters did you use to get this result (43.3 instance segmentation mAP)? How much is the learning rate? How many iterations? How many images per batch? how many epochs?

Some question about find the minimum distance between virtual and real points

Hi, I go over your code in details recently, but in function add_virtual_mask, i find it compute the distance between a virtual point and all real point then select the minimum, ignore if the real point is in the same camera_id and same mask with the virtual point. I wonder that's a little unreasonable? Because it may use a real point's depth which is in other mask or even in other camera_id. Looking forward to your reply~

some issues about process of generate virtual points

Firstly, thank you for providing the source code.
I want to replicate the process of generating virtual points from you, but I have encountered some difficulties.
I have installed the code according to the steps you provided and executed it on the terminal:

python tools/create_data.py nuscenes_data_prep --root_path=NUSCENES_TRAINVAL_DATASET_ROOT --version="v1.0-trainval" --nsweeps=10 --virtual True

This section generates two empty virtual point cloud folders and infos_val_10sweeps_withvelo_filter_True.pkl and infos_train_10sweeps_withvelo_filter_True.pkl file

Execute again:
python virtual_gen.py --info_path data/nuScenes/infos_train_10sweeps_withvelo_filter_True.pkl MODEL.WEIGHTS centernet2_checkpoint.pth

He reminded me: No such file or directory: 'home/zcj/CenterPoint/data/nuScenes/samples/LIDAR_TOP_VIRTUAL/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402927647951.pcd.bin.pkl.npy'

I guess I may need to train an image semantic segmentation result first, but I couldn't find any relevant training in your code. Do I need to provide it myself? I also couldn't find a place in the previous code to insert the training results. At the same time, I noticed that I downloaded a pth file while executing the second code mentioned above. What is the purpose of this file.

Also, I am using the mini dataset, and I am not sure if this is the reason

I am a freshman with a limited understanding of this research field and a serious lack of code proficiency. If you could provide more detailed operational procedures, I would greatly appreciate it.

首先，谢谢您提供的开原代码。
我想要复现您生成虚拟点的过程，但是我遇到了一些麻烦。
我按照您给的步骤安装好了代码，并在终端执行了：
python tools/create_data.py nuscenes_data_prep --root_path=NUSCENES_TRAINVAL_DATASET_ROOT --version="v1.0-trainval" --nsweeps=10 --virtual True

这一段生成了两个空的虚拟点云文件夹，和infos_val_10sweeps_withvelo_filter_True.pkl与infos_train_10sweeps_withvelo_filter_True.pkl文件。

再执行了：
python virtual_gen.py --info_path data/nuScenes/infos_train_10sweeps_withvelo_filter_True.pkl MODEL.WEIGHTS centernet2_checkpoint.pth

这段他提示 No such file or directory: 'home/zcj/CenterPoint/data/nuScenes/samples/LIDAR_TOP_VIRTUAL/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402927647951.pcd.bin.pkl.npy'

我猜测我可能需要先训练一个图片语义分割结果来，但在您的代码中没有找到相关的训练，这是需要我自己提供吗，我也没在前面代码中找到可以插入训练结果的地方，同时我注意到在执行上面提到的第二个代码时下载了一个pth文件，请问这个文件的作用是什么。

另外我用的是mini数据集，不知道会不会是这个原因

我是一个研一新生，对这个研究领域一知半解，代码水平也严重不足，如果作者能够提供更加详细的操作过程，我将万分感谢。

How long do you need to train MVP?

Thanks for your work! But it takes me about 3 days to train MVP (with 4 V100).
I changed your env to spconv 2 and CUDA11.1. If using CUDA10.2 and spconv 1 mentioned in your install guide, it takes longer.
So could you please tell me how long do you take?
I have enough GPU memories and is it possible to accelerate the training? Thanks!

WBF

Could you please share your code of "weighted boxes fusion"?

"RuntimeError: CUDA error: device-side assert triggered" When running virtual_gen.py with nuscenev1.0-mini

Thanks for the wonderful job!
However, I encountered this error when I tried to generate virtual points:

[04/12 15:03:46 detectron2]: Full config saved to ./output/CenterNet2/nuImages_CenterNet2_DLA_640_8x/config.yaml
[04/12 15:03:46 d2.utils.env]: Using a generated random seed 46522022
Loading pretrained
[04/12 15:03:55 fvcore.common.checkpoint]: [Checkpointer] Loading from centernet2_checkpoint.pth ...
0%| | 0/2340 [00:00<?, ?it/s]/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [96,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [97,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [98,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [99,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [0,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [1,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [2,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [3,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [4,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [5,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [6,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [7,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [10,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [11,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [12,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [13,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [14,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [15,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [16,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [17,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [18,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [19,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [24,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [25,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [26,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [27,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [28,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [29,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [32,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [33,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [34,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [35,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [36,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [37,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [38,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [39,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [40,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [41,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [42,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [43,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [44,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [45,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [46,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [47,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [48,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [49,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [50,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [51,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [52,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [53,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [54,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [55,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [56,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [57,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [62,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [63,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [64,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [65,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [66,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [67,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [68,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [69,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [70,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [71,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [72,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [73,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [74,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [75,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [76,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [77,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [78,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [79,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [80,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [81,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [82,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [83,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [84,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [85,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [86,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [87,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [88,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [89,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [90,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [91,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [92,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [93,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [94,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [95,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
0%| | 1/2340 [00:02<1:38:43, 2.53s/it]
Traceback (most recent call last):
File "virtual_gen.py", line 311, in
main(args)
File "virtual_gen.py", line 271, in main
res = process_one_frame(info, predictor, data)
File "/root/anaconda3/envs/torch/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "virtual_gen.py", line 238, in process_one_frame
intrinsics=to_batch_tensor(all_cams_intrinsic), transforms=to_batch_tensor(all_cams_from_lidar))
File "/root/anaconda3/envs/torch/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "virtual_gen.py", line 92, in add_virtual_mask
indices =mask.reshape(W, H).nonzero()
RuntimeError: CUDA error: device-side assert triggered

Could you please give me some suggestions?

Doubts about virtual_lidar_points

Hi, thank you for such a great job. Some questions need your guidance. I downloaded the virtual points() generated by you. What I want to confirm is that each file here is the same as the point cloud bin file provided in nuscenes, that is, the point cloud information under the laser time stamp. I want to know what is the information length of each laser point and what information does each index correspond to?

generate the points

Thank you for the great work.
when I generate the points with another detector with the:

python virtual_gen.py --info_path data/nuScenes/infos_train_10sweeps_withvelo_filter_True.pkl MODEL.WEIGHTS checkpoint.pth

I get this error:

8%|███████▊ | 21981/271572 [00:03<00:25, 9734.81it/s]/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [10,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [11,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [12,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [13,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [14,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [15,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [16,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [17,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [18,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [19,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. 8%|███████▊ | 22158/271572 [00:04<00:54, 4589.35it/s] Traceback (most recent call last): File "virtual_gen.py", line 314, in <module> main(args) File "virtual_gen.py", line 277, in main res = process_one_frame(info, predictor, data) File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context return func(*args, **kwargs) File "virtual_gen.py", line 227, in process_one_frame pred_label, score, pred_mask = postprocess(result[camera_id]) File "virtual_gen.py", line 197, in postprocess labels = labels[~empty_mask] RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

what could be the problem?

what is the shape of the virtual points dimension?

hi~

I wonder if using CenterPoint changes the input part of the model? Does the virtual point have dimensions other than (x,y,z,r)? If I want to use the kitti data set, and the kitti data set has input (x,y,z,r), does that mean that I need to change the input part of the detection model

Generating results on nuscenes test set

Hi,
I am wondering how can I generate virtual points for nuscenes test set. Since I don't find a .pkl file like "infos_train_10sweeps_withvelo_filter_True.pkl", which seems to be required for generating virtual points with command

python virtual_gen.py --info_path data/nuScenes/infos_train_10sweeps_withvelo_filter_True.pkl  MODEL.WEIGHTS centernet2_checkpoint.pth 

Where is the difference between creating data with or without virtual point?

Hi! Thank you for your great work!

I am following the instructions in getting_started. However, I fail to understand the difference between creating data with virtual points (MVP) and creating data without virtual points (CenterPoint Baseline). Based on the instructions, the command for MVP is

# nuScenes
python tools/create_data.py nuscenes_data_prep --root_path=NUSCENES_TRAINVAL_DATASET_ROOT --version="v1.0-trainval" --nsweeps=10 --virtual True 

And the command for CenterPoint is

# nuScenes
python tools/create_data.py nuscenes_data_prep --root_path=NUSCENES_TRAINVAL_DATASET_ROOT --version="v1.0-trainval" --nsweeps=10 --virtual False 

It seems to me that the only difference is --virtual, which is only used in creating gt sampling here. But according to the instructions, the improvements of gt sampling on nuscenes is marginal and thus can be simply discarded. If that is the case, it seems to me there is no real difference between creating data with or without virtual points.

Am I missing something here? Thanks!

questions

May i ask questions again?

1- where does the z axis value for the generated virtual points come from? It is calculated by the interpolated depth ? right?

2- virtual points are generated from the instance masks and not from Frustum? right?

The paper says: "We start by randomly sampling 2D points s 2 m from each instance mask m."

but it also says:
"we generate virtual points from each frustum Fj ."
and:
"We augment sparse Lidar point cloud with dense semantic virtual points generated from 2D detections".

I think the first sentence is the exact sentence.

3- Why is dist_thresh=3000 chosen? Based on what?

4- Will virtual points be generated even if there is only one lidar point in instance mask?

5- How to take class scores of the virtual points after entering the real and virtual points into the 3d detector? Is there a certain percentage that is taken, for example, 50% of the 2d detector score and 50% of the 3d detector score?

6- For example, if there is a mask for a distant car, one lidar point is projected within it and 50 virtual lidar points are generated with a 50% score, that's a car. How would the final result be calculated in the 3d detector?

7- how are the virtual points unprojected into 3D? where does the inverse matrix transform come from? from the neighbor lidar points from which the depth is taken?

8- What is painted points here?
https://github.com/tianweiy/CenterPoint/blob/8fcdc944bbb455bd25943b331aaf961aa0ab32cd/det3d/models/readers/dynamic_voxel_encoder.py#L28

Question for global_scaling_v2

Please see https://github.com/tianweiy/CenterPoint/blob/2145ada1f96b7b4b8f19f3b0012aab589962806f/det3d/core/sampler/preprocess.py#L838

def global_scaling_v2(gt_boxes, points, min_scale=0.95, max_scale=1.05):
    noise_scale = np.random.uniform(min_scale, max_scale)
    points[:, :3] *= noise_scale
    gt_boxes[:, :-1] *= noise_scale
    return gt_boxes, points

"gt_boxes" is [box_number,9]. 9 corresponds to x, y, z, w, h, l, vx, vy, r.
So why is the vx and vy of gt_boxes scaled? Is this a bug?
And by the way, where can I find the data format description of ground truth annotations (the definition of each dimension)? I can't find it in NuScenes' website and github. Thanks!

Can it be applied to the waymo dataset?

I'm using centerpoint using waymo dataset.
I haven't used the nuscence dataset
Is it difficult to apply MVP to the waymo dataset?
Is it possible to just use it as a different parameter value?

semantic information different between PointPainting and MVP

I read the code of pointpainting which is based openpcdet.It use segmentation score which is added to one to be paint point semantic information. But virtual_gen.py use one-hot encoded and classification confidence as semantic information.Is my understanding of the code right? Which is better and why?

question about MaskFormer pretrained model.

Hi! Thank you for your work!

1. I'm trying to reproduce the results of MVP-KITTI in your paper, could you provide more information about the pretrained MaskFormer model? Which one should I choose to get the same result as you did.
2. Also, there are some slight differences in results trained with PointPillars and KITTI. I used PointPillars in CenterPoint-KITTI which has the same config file as the one in your OpenPCdet. All settings are default. Also I download the pretrained model in https://github.com/tianweiy/OpenPCDet (model-18M) and get the results as below. The download model gets the same results as documented in your web but I record the Car_R40_moderate_0.7 instead of Car_moderate_0.7 so I can compare it with my result.

Method | Car@R40_moderate | Cyclist@R40_moderate | Pedestrian@R40_moderate
PointPillars(my training) | 77.4143 | 64.2696 | 50.4214
PointPillars(download) | 78.3964 | 62.8074 | 51.4145
PointPillars(in paper) | 77.3 | 62.7 | 52.3

How can I get the results in your paper? Are there any changes I need to make?

Generage `data/nuScenes/infos_train_10sweeps_withvelo_filter_painted_True.pkl` file

Hi, thanks for your excellent work.
I have trained one-stage model nusc_centerpoint_voxelnet_0075voxel_fix_bn_z_scale_virtual.py. But after running python -m torch. distributed. launch --nproc_per_node=3 ./tools/train. py configs/mvp/nusc_two_stage_base_with_virtual. py, the error occurs"

 FileNotFoundError: [ Errno 2] No such file or directory: ' data/nuScenes/infos_tra in_10sweeps_withvelo_filter_painted_True. pkl"

And I train step is followed by https://github.com/tianweiy/MVP/blob/main/README.md one step by step.
But exactly, the file infos_tra in_10sweeps_withvelo_filter_painted_True. pkl is not generated.
My data folder is organized as follows：

So, what step is lacking? Thank you very much.

MVP reproduction on NuScenes with OpenPCDet (mAP:64.22 NDS:68.96)

Hi! Thank you for your great work!
I'm trying to reproduce MVP(CenterPoint-VoxelNet) results with OpenPCDet, but the results seems not good enough compared to your official results (mAP: 66, NDS:69.9). I modify points loading related functions and points dimensions (5 to 22) related code just like yours. I check the voxel features and nothing seems wrong to me. Could it be test time augmentation that leads to this inferior results? It seems that there's no TTA in default config file of CenterPoint in OpenPCDet. Did you get the result of mAP 66 with TTA?

Missing sweeps virtual points

Hi, I just found there are about 2000 sweeps of virtual points missing.

Here are the number of files in sweeps/LIDAR_TOP/

ls sweeps/LIDAR_TOP/ | wc -l
297737

Here are the number of files in sweeps/LIDAR_TOP_VIRTUAL/

ls data/nuscenes/sweeps/LIDAR_TOP_VIRTUAL/ | wc -l
295489

I directly download and extract the data from the official link here. I am pretty sure I didn't mess with the data. I wonder is this normal?

Thanks!

How to implement CenterPoint + Ours(w/o virtual)

May I ask how to implement CenterPoint + Ours(w/o virtual)? Is it just to delete virtual_points2 in points = np.concatenate([points, virtual_points1, virtual_points2], axis=0).astype(np.float32) in MVP-main/CenterPoint-vir/det3d/datasets/pipelines/loading.py read_file function?

PPT

Is it convenient to disclose the PPT in the video?

MaskFormer pretrained on coco-panoptic

Hi,
I notice that you generate instance masks for KITTI using the maskformer pretrained on coco-panoptic. I am wondering whether the domain gap between COCO and KITTI will lead to unsatisfied instance segmentation performances. Because I notice that the improvements on KITTI is not as drastic as on nuScenes.

How is the FPS of MVP?

I'm curious about the FPS of MVP. Is it much slower than pure centerpoint?

virtual points and points features alignment

Hi! Thank you for your work. I'm looking at the code and find that lidar points' features are not exactly aligned:
virtual_points1 = data_dict['real_points'][:, [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]]
virtual_points2 = data_dict['virtual_points']
points = np.concatenate([points, np.ones([points.shape[0], 15-num_point_feature])], axis=1)
virtual_points1 = np.concatenate([virtual_points1, np.zeros([virtual_points1.shape[0], 1])], axis=1)
virtual_points2 = np.concatenate([virtual_points2, -1 * np.ones([virtual_points2.shape[0], 1])], axis=1)
points = np.concatenate([points, virtual_points1, virtual_points2], axis=0).astype(np.float32)

So I wondered why not using orders like below (x, y, z, intensity(-1 for virtual points2), classification*10, score):
virtual_points1 = data_dict['real_points'][:, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]]
virtual_points2 = data_dict['virtual_points']
virtual_points2 = np.concatenate([virtual_points2[:, [0, 1, 2]] , -1 * np.ones([virtual_points2.shape[0], 1]), virtual_points2[:, [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]]], axis=1)

Split Voxelization? Two-stage?

I have two questions about the abation study in your paper:

1. What is split voxelization?

2. Does your repo contain the code about split voxelization or two-stage method?

generate the virtual points

Hello tianweiy,
when I generate the points with gpu RTX 3090 , I get this by train set:
100%|█████████████████████████| 271572/271572 [110:45:35,<00:00, 1.46s/it]
and this by val set:
100%|█████████████████████████| 58066/58066 [25:44:38<00:00, 1.60s/it]

1- What does these numbers 271572,58066 mean? Is it the number of segmentation masks in the images?
num of images in val set is: 16445!

2- Why does it take so long to generate virtual points? Is this logical?

look help for visualization

Hi, i want to do some work about visualization, how to achieve the same result like figure 1 in the paper?

Some details about generating virtual points

Hello, thank you for open-sourcing the source code. I encountered some problems during the learning process and would like to ask you for advice. I saw that in the code, N virtual points are randomly generated in the instance mask, and the index is obtained by the nearest neighbor distance and distance threshold, but the depth value is directly given by the original point cloud. Is it fixed? I don't understand here.

CenterPoint+PointPainting on nuScenes Val dataset

Hi! Thanks for your paper MVP.
But I have a question about the result of CenterPoint+PointPainting on nuScenes Val dataset.
As you mentioned in the paper, it reaches 66.5 overall mAP. But my result is only 65.4.
I train the model by just throwing away those virtual points since you have already painted those real points on the masks.
All other configurations are the same with https://github.com/tianweiy/CenterPoint/blob/master/configs/mvp/nusc_centerpoint_voxelnet_0075voxel_fix_bn_z_scale_virtual.py

Could you please tell me the reason? Thanks!

Is there an online implementation of MVP?

Hi, thanks for your excellent work.

In my understanding, the released code is the offline implementation, since the virtual point is preprocessed before the model is trained and tested.

I am curious about whether an online implementation has been tried, where the virtual point is added to the original point cloud in the process of training and testing?

Thanks in advance.

question about centernet2

hello, tianweiy! Can you share how you train the model Centernet2, and how about the performance, like miou ....Thankyou!

Where is the code of ground truth sampling(GT-Paste)?

Could you kindly indicate the specific location of the code snippet of ground truth sampling(GT-Paste)?

How did you visualize the pcd.bin.pkl.npy files of virtual points?

Thank you for your great job.
I want to visualize them and then compare with original points.
Can i have some tips?

nuImages 2D annotation aligned with nuScenes camera data

Thanks for your great work and released code!
I am wondering if there are any methods to align the camera images in nuScences with those in nuImages. Since I observed that roughly 76% virtual points, selected according to CenterNet2's instance segmentation masks, are discarded because that they don't correspond to any instances annotated in the nuScenes. Hence, I am wondering the influence of the discrepancy between the instances detected by CenterNet2 and those annotated in the nuScenes, and would like to know if there are any possible ways to directly obtain the 2D annotations of camera images within the nuScenes?

No module named 'CenterNet2'

I have installed Detectron2 , CenterPoint and CenterNet2, But while running virtual_gen.py in the mvs root folder I get this error "No module named 'CenterNet2'.

I have already added the paths of CenterPoint and CenterNet2.

question about kitti

could you release the kitti code？

good job!! l really like it

I found that the core of generating virtual points in the article lies in using 2D masks to achieve random sampling enhancement for 3D point cloud in a simple and effective way. At the same time, I have observed some issues in my experiments. I found that objects in Lidar modal data such as cars and pedestrians seem to be more sensitive to edge contours, enhancing the continuity and integrity of the contours, making the shape of Lidar more prominent.

This seems to bring greater improvement for 3D detection. In the future, I plan to delve deeper into researching and transforming the ideas from your paper into my own. Thank you for your work!

Sorry

setting of "num_virtual"

Did you conduct experiments for different setting of the param "num_virtual"?
Could you explain the thinking why sets 'num_virtual = 50' ? Why not larger num_virtual, eg. num_virtual = 100, 200.

Thanks!