dvlab-research / largekernel3d Goto Github PK

Dear Author, I train a detection model with config "nusc_centerpoint_voxelnet_0075voxel_fix_bn_z_largekernel3d_large.py" with 8GPUs，the model performance is lower than reported. Do you know what's wrong with my trained model?

performance detail:
mAP: 0.5944
mATE: 0.2902
mASE: 0.2516
mAOE: 0.3343
mAVE: 0.2870
mAAE: 0.1911
NDS: 0.6618
Eval time: 104.5s

Per-class results:
Object Class AP ATE ASE AOE AVE AAE
car 0.851 0.181 0.153 0.110 0.267 0.193
truck 0.563 0.331 0.180 0.116 0.255 0.227
bus 0.707 0.333 0.177 0.073 0.487 0.268
trailer 0.393 0.500 0.201 0.564 0.221 0.183
construction_vehicle 0.199 0.713 0.434 1.025 0.123 0.312
pedestrian 0.845 0.147 0.272 0.389 0.219 0.099
motorcycle 0.583 0.202 0.238 0.240 0.499 0.231
bicycle 0.415 0.162 0.268 0.414 0.225 0.016
traffic_cone 0.690 0.136 0.322 nan nan nan
barrier 0.698 0.198 0.270 0.077 nan nan
Evaluation nusc: Nusc v1.0-trainval Evaluation
car Nusc dist [email protected], 1.0, 2.0, 4.0
75.43, 85.77, 88.99, 90.23 mean AP: 0.8510555593669333
truck Nusc dist [email protected], 1.0, 2.0, 4.0
37.71, 54.44, 64.86, 68.22 mean AP: 0.5630814131474254
construction_vehicle Nusc dist [email protected], 1.0, 2.0, 4.0
3.41, 12.18, 27.07, 37.13 mean AP: 0.19945290009625813
bus Nusc dist [email protected], 1.0, 2.0, 4.0
46.04, 69.87, 82.21, 84.80 mean AP: 0.7072839081535547
trailer Nusc dist [email protected], 1.0, 2.0, 4.0
11.06, 35.42, 50.44, 60.39 mean AP: 0.39329063939703013
barrier Nusc dist [email protected], 1.0, 2.0, 4.0
59.75, 70.12, 73.86, 75.32 mean AP: 0.6976273792134366
motorcycle Nusc dist [email protected], 1.0, 2.0, 4.0
52.38, 59.28, 60.53, 61.11 mean AP: 0.5832459929266106
bicycle Nusc dist [email protected], 1.0, 2.0, 4.0
40.47, 41.59, 41.82, 41.98 mean AP: 0.41463017310991623
pedestrian Nusc dist [email protected], 1.0, 2.0, 4.0
82.15, 83.86, 85.29, 86.59 mean AP: 0.8447317287125544
traffic_cone Nusc dist [email protected], 1.0, 2.0, 4.0
66.22, 67.69, 69.40, 72.53 mean AP: 0.6896184124816187

Hi, I'm interested in your work. I looked at your paper and code and I have a question. Regarding Figure 3, in the paper you replaced the normal convolutional layer with SW__LK, 7X7 at training time and 3X3 at inference time, is that right? And the convolutional layer corresponding to spatialgroupconv and spatialgroupconvV2 in the code for training and inference stages respectively? Or should we just use spatialgroupconvV2 for training and inference?

Hello, thank you for your excellent work,
may I ask how you draw this SFD visualization diagram?

Hi, thanks for your work, I find that depth-wise conv is used. But this feature is not implemented in spconv and MinkowskiNet. Did you implement it yourself? Will the code of depth-wise conv be open?

hi ! I'm interesting in your work, but there is one thing I don't quite understand. Can you tell me how many groups other large kernels need to be divided into and how they are divided (eg. spatial-wise 7×7×7 convolutions is grouped into 3×3×3 splits.)

I have noticed that you also implement largeKernel3D for Minkowski Engine, but only sparse conv largeKernel3D I can find. I wonder whether you will give the source code for largerKernel3D in Minkowski Engine? Thanks!

Hi, thanks for your work.

Regarding the reformulation from large kernels into small ones, Could you please explain the reformulation process in detail, I cannot find the introduction to this process in the paper.

Looking forward to your reply! Many thanks!

How did you validate runtime of your model??
Thank you

Dear Author, I am training detection model with config "nusc_centerpoint_voxelnet_0075voxel_fix_bn_z_largekernel3d_large,py" in two V100 GPUs. The conv type is 'spatialgroupconvv2'. It seems that it will cost 12days to train the model with 20 epochs. When trained with four V100 GPUs, the cost time is also 12 days. Is this normal？It will be very nice if you can share your traning logs or share your training time and gpu numbers.

here is my traing log with 2 GPUs:
2023-05-08 19:00:44,415 - INFO - Epoch [1/20][510/30895] lr: 0.00010, eta: 11 days, 17:20:45, time: 1.361, data_time: 0.121, transfer_time: 0.019, forward_time: 0.327, loss_parse_time: 0.001 memory: 7336,

here is my traing log with 4 GPUs:
2023-05-09 10:22:58,283 - INFO - Epoch [1/20][100/15448] lr: 0.00010, eta: 14 days, 20:59:34, time: 3.149, data_time: 0.254, transfer_time: 0.015, forward_time: 0.333, loss_parse_time: 0.000 memory: 6913,

Hello, I encountered the following error while using the Det3D code for training. How can I solve it?

2023-06-16 15:43:45,501 - INFO - Start running, host: root@autodl-container-28d9119efa-79377d73, work_dir: /root/FocalsConv-master/CenterPoint/work_dirs/CONFIG
2023-06-16 15:43:45,501 - INFO - workflow: [('train', 1)], max: 20 epochs
2023-06-16 15:43:51,726 - INFO - finding looplift candidates
2023-06-16 15:43:51,755 - INFO - finding looplift candidates
2023-06-16 15:43:51,948 - INFO - finding looplift candidates
2023-06-16 15:43:52,046 - INFO - finding looplift candidates
2023-06-16 15:43:52,075 - INFO - finding looplift candidates
2023-06-16 15:43:52,119 - INFO - finding looplift candidates
[Exception|indice_conv|subm]feat=torch.Size([135499, 16]),w=torch.Size([7, 7, 7, 16, 16]),pair=torch.Size([2, 343, 135499]),pairnum=tensor([ 4799, 9601, 11036, 4431, 3988, 3099, 2711, 3610, 6719, 13681,
8034, 4572, 4015, 2955, 2820, 4329, 11488, 13846, 5758, 4371,
3113, 2780, 3744, 7171, 16534, 9582, 4290, 3497, 2950, 3748,
4348, 10300, 13840, 5414, 3772, 2833, 3077, 3417, 4984, 13013,
9209, 4316, 2435, 2449, 2808, 3222, 7937, 12206, 5862, 5213,
9682, 10098, 4573, 4173, 3384, 2996, 4151, 7365, 13003, 7792,
5035, 4334, 3385, 3423, 5119, 12062, 13400, 6321, 4835, 3552,
3306, 4584, 8450, 16577, 9596, 4787, 3708, 3381, 4466, 5524,
11318, 13242, 5704, 3935, 3263, 3694, 4214, 6031, 12766, 8758,
4469, 2813, 2798, 3321, 3851, 8373, 11029, 5845, 5048, 11281,
11257, 5583, 5197, 4622, 4496, 4608, 8173, 15342, 9037, 6700,
6068, 5341, 4580, 6133, 14271, 15437, 8596, 7319, 5876, 4787,
6313, 11034, 20252, 11914, 7667, 6330, 5312, 6715, 8446, 15682,
15897, 8205, 7022, 5608, 6370, 7243, 9905, 17199, 11063, 7679,
5535, 5803, 6578, 7338, 13694, 15088, 9110, 42222, 50556, 46215,
36413, 35958, 36707, 36908, 38101, 47028, 56527, 43580, 41221, 41523,
39315, 37325, 43368, 62136, 65413, 52124, 45853, 40064, 38684, 45057,
64356, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0], device='cuda:0', dtype=torch.int32),act=135499,algo=ConvAlgo.Native
SPCONV_DEBUG_SAVE_PATH not found, you can specify SPCONV_DEBUG_SAVE_PATH as debug data save path to save debug data which can be attached in a issue.
Traceback (most recent call last):
File "./tools/train.py", line 146, in
main()
File "./tools/train.py", line 134, in main
train_detector(
File "/root/FocalsConv-master/CenterPoint/det3d/torchie/apis/train.py", line 331, in train_detector
trainer.run(data_loaders, cfg.workflow, cfg.total_epochs, local_rank=cfg.local_rank)
File "/root/FocalsConv-master/CenterPoint/det3d/torchie/trainer/trainer.py", line 553, in run
epoch_runner(data_loaders[i], self.epoch, **kwargs)
File "/root/FocalsConv-master/CenterPoint/det3d/torchie/trainer/trainer.py", line 419, in train
outputs = self.batch_processor_inline(
File "/root/FocalsConv-master/CenterPoint/det3d/torchie/trainer/trainer.py", line 378, in batch_processor_inline
losses = model(example, return_loss=True)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/FocalsConv-master/CenterPoint/det3d/models/detectors/voxelnet_focal.py", line 85, in forward
x, _, loss_box_of_pts = self.extract_feat(data, batch_dict)
File "/root/FocalsConv-master/CenterPoint/det3d/models/detectors/voxelnet_focal.py", line 38, in extract_feat
x, voxel_feature, loss_box_of_pts = self.backbone(
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/FocalsConv-master/CenterPoint/det3d/models/backbones/scn_largekernel.py", line 350, in forward
x_conv1, _loss = self.conv1(x, batch_dict)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/FocalsConv-master/CenterPoint/det3d/models/backbones/scn_largekernel.py", line 44, in forward
input = module(input)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/FocalsConv-master/CenterPoint/det3d/models/backbones/scn_largekernel.py", line 215, in forward
out = self.conv1(x)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/FocalsConv-master/CenterPoint/det3d/models/backbones/scn_largekernel.py", line 127, in forward
x_conv_block = self.block(x_conv)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/spconv/pytorch/conv.py", line 741, in forward
return self._conv_forward(self.training, input, self.weight, self.bias, add_input,
File "/root/miniconda3/lib/python3.8/site-packages/spconv/pytorch/conv.py", line 312, in _conv_forward
out_features = Fsp.indice_subm_conv(
File "/root/miniconda3/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 94, in decorate_fwd
return fwd(*args, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/spconv/pytorch/functional.py", line 327, in forward
raise e
File "/root/miniconda3/lib/python3.8/site-packages/spconv/pytorch/functional.py", line 308, in forward
return ops.indice_conv(features,
File "/root/miniconda3/lib/python3.8/site-packages/spconv/pytorch/ops.py", line 860, in indice_conv
ConvGemmOps.indice_conv(alloc, ext_mm, GEMM_CPP, ALL_WEIGHT_IS_KRSC,
RuntimeError: /tmp/pip-build-env-oq41cytq/overlay/lib/python3.8/site-packages/cumm/include/tensorview/tensor.h(770)
stride_valid assert faild. non-contiguous stride can't handled.

Hi, I'm interested in your great work and I'm trying to run the semantic segmentation experiment. But after preparing ScanNetV2 data following the instructions of PointGroup(https://github.com/dvlab-research/PointGroup) as your instructions say, I still cannot launch the training process. After the preprocessing, I get files like these

but the code seems to only support [scene_id].ply files. How to solve this problem? Any other preprocessing needed?

Here is my log with a bug, how can I solve this problem?

[Exception|indice_conv|subm]feat=torch.Size([137566, 16]),w=torch.Size([7, 7, 7, 16, 16]),pair=torch.Size([2, 343, 137566]),pairnum=tensor([ 5001,  4071,  2915,  2876,  2559,  2547,  2521,  4382,  5716,  4867,
         3664,  3167,  3121,  2835,  3526,  5018,  7021,  6005,  4319,  3926,
         3575,  3916,  4679,  6829,  9400,  7201,  5169,  4423,  4106,  4546,
         5253,  7129,  7777,  5756,  3978,  3274,  3504,  3874,  4283,  5309,
         6038,  4567,  2878,  2974,  3229,  3410,  3441,  4449,  5227,  4487,
         3883,  3429,  3430,  3017,  2970,  2785,  4539,  5350,  4998,  4443,
         3730,  3469,  3128,  4072,  5174,  6710,  6427,  4945,  4201,  3716,
         4024,  4918,  6864,  8956,  6847,  5047,  4072,  3896,  4419,  5554,
         7007,  7141,  5674,  4268,  3469,  3878,  4567,  4992,  5347,  5751,
         4694,  3185,  3406,  3888,  4043,  3898,  4522,  4774,  8162,  7692,
         7195,  6938,  6410,  6041,  5768,  8543,  8755,  8271,  7939,  7052,
         6395,  5980,  8122,  9124, 10307, 10506,  8607,  7326,  6615,  8280,
         9333, 11303, 13434, 10871,  8458,  6993,  7694,  8293,  9598, 10743,
         9695,  7938,  6372,  6528,  6814,  7356,  7580,  7401,  7089,  6270,
         5683,  5811,  6181,  6397,  5909,  5726,  5781, 43530, 43224, 40874,
        39942, 38939, 38032, 36751, 43545, 50091, 50867, 47721, 45472, 43093,
        39536, 44726, 52853, 62406, 66190, 56487, 49107, 44248, 47333, 54066,
        66982,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
            0,     0,     0], device='cuda:0', dtype=torch.int32),act=137566,algo=ConvAlgo.Native
no apex
No Tensorflow
SPCONV_DEBUG_SAVE_PATH not found, you can specify SPCONV_DEBUG_SAVE_PATH as debug data save path to save debug data which can be attached in a issue.
Traceback (most recent call last):
  File "SpMiddleResNetFHDLargeKernel_forward.py", line 13, in <module>
    ret, multi_scale_voxel_features,_ = center_net(voxel_features, batch_dict, voxel_coords, batch_size, dense_shape)
  File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/dl-01/HDD/code/FocalsConv/CenterPoint/det3d/models/backbones/scn_largekernel.py", line 290, in forward
    x_conv1 = self.conv1(x)
  File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/spconv/pytorch/modules.py", line 138, in forward
    input = module(input)
  File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/dl-01/HDD/code/FocalsConv/CenterPoint/det3d/models/backbones/scn_largekernel.py", line 181, in forward
    out = self.conv1(x)
  File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/dl-01/HDD/code/FocalsConv/CenterPoint/det3d/models/backbones/scn_largekernel.py", line 92, in forward
    x_conv_block = self.block(x_conv)
  File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/spconv/pytorch/conv.py", line 755, in forward
    return self._conv_forward(self.training,
  File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/spconv/pytorch/conv.py", line 327, in _conv_forward
    out_features = Fsp.indice_subm_conv(
  File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 118, in decorate_fwd
    return fwd(*args, **kwargs)
  File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/spconv/pytorch/functional.py", line 327, in forward
    raise e
  File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/spconv/pytorch/functional.py", line 308, in forward
    return ops.indice_conv(features,
  File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/spconv/pytorch/ops.py", line 860, in indice_conv
    ConvGemmOps.indice_conv(alloc, ext_mm, GEMM_CPP, ALL_WEIGHT_IS_KRSC,
RuntimeError: /tmp/pip-build-env-h7b1kdx5/overlay/lib/python3.8/site-packages/cumm/include/tensorview/tensor.h(770)
stride_valid assert faild. non-contiguous stride can't handled.

Hi, thanks for your released code! I noticed that you released your last work FocalsConv based on both det3d and OpenPCDet, but this time you only released the version on det3d. Do you have any plan to add LargeKernel3D to OpenPCDet? Thanks!