edwardzhou130 / panoptic-polarnet Goto Github PK

Hi!

Thank you for your wonderful work!

Could you please share the code you used to visualize the predictions of the pretrained network to obtain the file https://github.com/edwardzhou130/Panoptic-PolarNet/blob/main/imgs/Visualization.gif?

Thank you!

I notice that in the prediction results there are these very big labels.
I load the label file (predicted result) with

label = np.fromfile(label_path, dtype=np.uint32, count=-1).reshape([-1,1])

{'model_name': 'Panoptic_PolarNet', 'dataset': {'name': 'semantickitti', 'path': 'data', 'output_path': 'out/SemKITTI', 'instance_pkl_path': 'data', 'rotate_aug': True, 'flip_aug': True, 'inst_aug': True, 'inst_aug_type': {'inst_os': True, 'inst_loc_aug': True, 'inst_global_aug': True}, 'gt_generator': {'sigma': 5}, 'grid_size': [480, 360, 32]}, 'model': {'model_save_path': './Panoptic_SemKITTI.pt', 'pretrained_model': 'empty', 'polar': True, 'visibility': True, 'train_batch_size': 2, 'val_batch_size': 2, 'test_batch_size': 1, 'check_iter': 4000, 'max_epoch': 100, 'post_proc': {'threshold': 0.1, 'nms_kernel': 5, 'top_k': 100}, 'center_loss': 'MSE', 'offset_loss': 'L1', 'center_loss_weight': 100, 'offset_loss_weight': 10, 'enable_SAP': True, 'SAP': {'start_epoch': 30, 'rate': 0.01}}}
Using MSE for heatmap regression, weight: 100
Using L1 for offset regression, weight: 10
[PANOPTIC EVAL] IGNORE: [0]
[PANOPTIC EVAL] INCLUDE: [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19]
0%| | 0/9565 [00:00<?, ?it/s]Traceback (most recent call last):
File "/data/p202008021038/Panoptic-PolarNet-main/train.py", line 236, in
main(args)
File "/data/p202008021038/Panoptic-PolarNet-main/train.py", line 106, in main
for i_iter,(train_vox_fea,train_label_tensor,train_gt_center,train_gt_offset,train_grid,,,train_pt_fea) in enumerate(train_dataset_loader):
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in next
data = self._next_data()
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
return self._process_data(data)
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
data.reraise()
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/_utils.py", line 428, in reraise
raise self.exc_type(msg)
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
data = fetcher.fetch(index)
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/data/p202008021038/Panoptic-PolarNet-main/dataloader/dataset.py", line 313, in getitem
xyz,labels,insts,feat = self.inst_aug.instance_aug(xyz,labels.squeeze(),insts.squeeze(),feat)
File "/data/p202008021038/Panoptic-PolarNet-main/dataloader/instance_augmentation.py", line 40, in instance_aug
random_choice = np.random.choice(len(self.instance_path[self.thing_list[n]]),count)
File "mtrand.pyx", line 903, in numpy.random.mtrand.RandomState.choice
ValueError: a must be greater than 0 unless no samples are taken

0%| | 0/9565 [00:00<?, ?it/s]

进程已结束，退出代码为 1

Hi. Really good work. I am trying to train the model but I can not debug why the losses are very high and PQ is too poor. Hope you may have some idea.

I notice that visualize.py is included and I have installed open3d. But I didn't get the visualized result by running test_pretrain.py, neither for validation sequence and test sequence. Can you suggest how to get it?

need author's help, please tell how to fix it.

If anybody has tested this repo on Nuscenes panoptic segmentaion please ping me. i will be glad if one can help. Thank you in advance.

Thank you for your response @edwardzhou130. I am trying as you told me but getting this error when I try to run dataloader. I have changed lidarseg to panopticseg in line 124

.

Hello, i implement your code on SemanticKITTI dataset, and keep the default config setting.

However, i have cost 3 days to train the network, and the current epoch is 30. The max_epoch is 100. Following this setting, i will cost 10 days to train the whole pipeline.

The GPU i used is Tesla-V100 16G.

The process is too slow to train, so any suggestions?

Thanks sincerely~

Congratulations for the great work you did. i am running the code you provided and I tested the pretrained model but why is it showing everything 0.00%. Dataloader is correct. While training it is also showing 0.00% at each iteration for validation check.

I am getting issue with generating the Panoptic labels for nuscenes in test_pretrain_nuscenes. Please help me.

Hi, thank you for your interesting paper and releasing the code. I got a question about panoptic fusion on NuScenes. What NMS parameters for panoptic fusion you use for baseline PolarNet + CBGS in table 3? I use group nms following the multi-group head and nms parameters per group is
nms_pre_max_size=1000, nms_post_max_size=83, nms_iou_threshold=0.2, score_threshold=0.1
But I got panoptic quality as 31 instead of 67 for PolarNet + CBGS. I got decent SQ, but for RQ there are a lot of false positives for things.

I followed the evaluation code here https://github.com/edwardzhou130/Panoptic-PolarNet/blob/main/utils/eval_pq.py and change min_points to 20 as stated in your paper.

Good work. Are you planning to upload a dataset_nuscenes and train_nuscenes files? I really want to test it.

Thank you for your work!
I ran two experiments to reproduce the results of the pretrained model listed in repository:

Results for the pretrained model:

Validation per class PQ, SQ, RQ and IoU:
car : 87.94% 91.18% 96.45% 93.65%
bicycle : 52.57% 77.49% 67.84% 46.37%
motorcycle : 65.87% 88.72% 74.24% 66.71%
truck : 62.52% 87.94% 71.10% 88.46%
bus : 52.99% 88.41% 59.93% 43.35%
person : 73.23% 89.91% 81.45% 62.23%
bicyclist : 85.31% 90.66% 94.11% 82.21%
motorcyclist : 41.43% 84.35% 49.12% 48.48%
road : 93.74% 93.74% 100.00% 93.63%
parking : 24.17% 70.29% 34.38% 45.05%
sidewalk : 76.44% 83.27% 91.80% 80.26%
other-ground : 0.00% 0.00% 0.00% 0.48%
building : 87.46% 90.54% 96.60% 90.28%
fence : 18.05% 70.02% 25.77% 51.68%
vegetation : 84.88% 85.51% 99.26% 86.44%
trunk : 45.25% 72.43% 62.47% 61.96%
terrain : 56.98% 74.62% 76.36% 73.54%
pole : 55.89% 73.23% 76.32% 62.35%
traffic-sign : 54.20% 73.82% 73.43% 46.23%

Current val PQ is 58.890
Current val miou is 64.387

Results of my experiments:

1. With initial parameters (training batch 2):

Validation per class PQ, SQ, RQ and IoU:
car : 85.57% 89.42% 95.70% 92.69%
bicycle : 45.07% 77.62% 58.06% 38.38%
motorcycle : 60.85% 87.75% 69.34% 53.97%
truck : 66.72% 89.95% 74.17% 73.40%
bus : 52.96% 88.94% 59.55% 35.49%
person : 69.85% 89.19% 78.32% 56.55%
bicyclist : 87.17% 91.88% 94.87% 77.88%
motorcyclist : 41.59% 80.88% 51.43% 50.53%
road : 92.43% 92.45% 99.98% 92.30%
parking : 22.38% 68.82% 32.52% 39.85%
sidewalk : 74.24% 80.88% 91.80% 77.54%
other-ground : 0.00% 0.00% 0.00% 0.30%
building : 86.14% 90.27% 95.43% 88.60%
fence : 17.57% 67.88% 25.89% 48.37%
vegetation : 83.11% 85.61% 97.08% 85.21%
trunk : 35.16% 70.44% 49.91% 53.74%
terrain : 54.49% 73.21% 74.43% 69.84%
pole : 55.85% 73.17% 76.33% 62.68%
traffic-sign : 52.02% 75.09% 69.28% 43.30%

Current val PQ is 57.010
Current val miou is 60.034

2. With initial parameters (training batch 1):
   Validation per class PQ, SQ, RQ and IoU:
   car : 87.62% 91.00% 96.29% 93.89%
   bicycle : 45.15% 77.46% 58.28% 42.56%
   motorcycle : 64.76% 89.65% 72.24% 64.91%
   truck : 54.76% 86.15% 63.56% 59.50%
   bus : 47.48% 89.10% 53.29% 37.47%
   person : 68.85% 89.77% 76.69% 54.36%
   bicyclist : 84.63% 89.93% 94.11% 82.65%
   motorcyclist : 32.75% 88.76% 36.89% 38.15%
   road : 93.18% 93.18% 100.00% 93.19%
   parking : 18.76% 66.70% 28.13% 40.71%
   sidewalk : 76.08% 81.98% 92.80% 79.75%
   other-ground : 0.00% 0.00% 0.00% 0.00%
   building : 86.66% 89.91% 96.39% 89.58%
   fence : 16.05% 67.93% 23.63% 49.75%
   vegetation : 82.70% 85.32% 96.93% 85.06%
   trunk : 42.30% 71.85% 58.88% 58.86%
   terrain : 53.89% 73.51% 73.31% 70.57%
   pole : 55.80% 72.88% 76.56% 62.47%
   traffic-sign : 51.75% 74.62% 69.36% 42.06%

Current val PQ is 55.956
Current val miou is 60.289

Both experiments were running for almost 100 epochs and their results (mIoU 60.3) don't match the results of your pretrained model (mIoU 64.3)

Could you please share the parameters and random seed for the experiment that achieved mIoU 64.3?

Thank you!

Hi. thank you for your last response. Now I am trying to deploy your model in one of my Jetson board with Velodyne sensor. First I want to try just feeding the single frame into the model. Can you suggest where can I change for input data in the test code. so I can easily generate predictions without giving data_path. My final goal is to have real-time segmentation with Lidar input. In more simplified way, input as a lidar raw data string and output as a predictions.

Hello, I am trying to run the code using command: python3 test_pretrain.py,

but return:
RuntimeError: Error(s) in loading state_dict for ptBEVnet:
size mismatch for BEV_model.network.inc.conv.0.weight: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.0.bias: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.0.running_mean: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.0.running_var: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.1.conv1.0.weight: copying a param with shape torch.Size([64, 32, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]).

in fact, I did not change the code of the paper.

Hi. I am trying to implement your code. I have input from Velodyne 64 as x,y,z,I. is it possible to use your model to directly use this values and obtain the predictions?

Thanks for your contribution and your work is very impressive and interesting. Just wondering about the check_occlusion function in the part of instance augmentation ( https://github.com/edwardzhou130/Panoptic-PolarNet/blob/main/dataloader/instance_augmentation.py#L160 ). The goal of this function is to check whether a point is within a (2 metre) range of any point in a point cloud right? Would dist = np.linalg.norm(points-center,axis=0) be dist = np.linalg.norm(points-center,axis=1) instead? When you compute norm on axis 0, it will be a shape of (3, ) for [x, y, z], and on axis 1 will be a shape of (N, ) for number of points.

Congratulations on such a good work. I have tried to replicate the results as shown in the paper but after training for >50 epochs, I am still not getting the results similar to the paper. I have used the batch size of 3, and everything similar to the config file in the git repo.
In another issue, you have mentioned to use around 50 epochs and sap as 20.

Following are my results:
Validation per class PQ, SQ, RQ and IoU: car : 87.58% 91.26% 95.96% 93.75% bicycle : 27.15% 75.09% 36.16% 18.39% motorcycle : 54.03% 89.00% 60.71% 55.05% truck : 69.47% 90.71% 76.59% 85.95% bus : 48.70% 90.19% 54.00% 43.03% person : 69.89% 90.11% 77.55% 56.81% bicyclist : 87.22% 91.86% 94.95% 76.11% motorcyclist : 1.67% 68.28% 2.44% 0.97% road : 93.33% 93.35% 99.98% 93.31% parking : 25.02% 71.31% 35.08% 44.85% sidewalk : 75.14% 82.17% 91.45% 78.90% other-ground : 0.00% 0.00% 0.00% 0.37% building : 86.55% 90.47% 95.66% 88.56% fence : 14.98% 67.74% 22.11% 46.67% vegetation : 82.53% 85.78% 96.22% 84.80% trunk : 39.25% 71.35% 55.00% 55.20% terrain : 53.21% 73.15% 72.73% 68.09% pole : 53.14% 72.97% 72.82% 60.64% traffic-sign : 50.63% 75.39% 67.16% 39.80% Current val PQ is 53.657 while the best val PQ is 55.571 Current val miou is 57.435

Can you please guide on where the performance gap can be reduced. It is urgent for our current work and help will be really appreciated.
Thank you.

I want to do inference on some sequence. Since the space on my server is limited and I cannot even download the 80G velodyne data zip, so I copied labels and velodyne data of sequence 00 from other's server and put it in data/sequences/00.
I run 'python test_pretrained.py'. But the results all showed 0.00.
My velodyne dir is : /data/sequences/00/velodyne. The labels dir is: /data/sequences/00/labels.
Part of the returned results are the follows:
'''
0it [00:00, ?it/s]
Current val PQ is 0.000
Current val miou is 0.000
Inference time per 1 is nan seconds, postprocessing time is nan seconds per scan
'''
Does anyone know if I put the data wrongly or something?
Also, is it possible that I do inference on raw point cloud data?

Hi!

Thank you for your wonderful work!

Do you have plans on releasing the model pretrained on the NuScenes dataset?

Thank you!

Hello, I am trying to run the code using command: python3 test_pretrain.py,

but return:
RuntimeError: Error(s) in loading state_dict for ptBEVnet:
size mismatch for BEV_model.network.inc.conv.0.weight: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.0.bias: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.0.running_mean: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.0.running_var: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.1.conv1.0.weight: copying a param with shape torch.Size([64, 32, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]).

in fact, I did not change the code of the paper.