edwardzhou130 / panoptic-polarnet Goto Github PK
View Code? Open in Web Editor NEWImplementation for Panoptic-PolarNet (CVPR 2021)
License: BSD 3-Clause "New" or "Revised" License
Implementation for Panoptic-PolarNet (CVPR 2021)
License: BSD 3-Clause "New" or "Revised" License
Hi!
Thank you for your wonderful work!
Could you please share the code you used to visualize the predictions of the pretrained network to obtain the file https://github.com/edwardzhou130/Panoptic-PolarNet/blob/main/imgs/Visualization.gif?
Thank you!
I notice that in the prediction results there are these very big labels.
I load the label file (predicted result) with
label = np.fromfile(label_path, dtype=np.uint32, count=-1).reshape([-1,1])
{'model_name': 'Panoptic_PolarNet', 'dataset': {'name': 'semantickitti', 'path': 'data', 'output_path': 'out/SemKITTI', 'instance_pkl_path': 'data', 'rotate_aug': True, 'flip_aug': True, 'inst_aug': True, 'inst_aug_type': {'inst_os': True, 'inst_loc_aug': True, 'inst_global_aug': True}, 'gt_generator': {'sigma': 5}, 'grid_size': [480, 360, 32]}, 'model': {'model_save_path': './Panoptic_SemKITTI.pt', 'pretrained_model': 'empty', 'polar': True, 'visibility': True, 'train_batch_size': 2, 'val_batch_size': 2, 'test_batch_size': 1, 'check_iter': 4000, 'max_epoch': 100, 'post_proc': {'threshold': 0.1, 'nms_kernel': 5, 'top_k': 100}, 'center_loss': 'MSE', 'offset_loss': 'L1', 'center_loss_weight': 100, 'offset_loss_weight': 10, 'enable_SAP': True, 'SAP': {'start_epoch': 30, 'rate': 0.01}}}
Using MSE for heatmap regression, weight: 100
Using L1 for offset regression, weight: 10
[PANOPTIC EVAL] IGNORE: [0]
[PANOPTIC EVAL] INCLUDE: [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19]
0%| | 0/9565 [00:00<?, ?it/s]Traceback (most recent call last):
File "/data/p202008021038/Panoptic-PolarNet-main/train.py", line 236, in
main(args)
File "/data/p202008021038/Panoptic-PolarNet-main/train.py", line 106, in main
for i_iter,(train_vox_fea,train_label_tensor,train_gt_center,train_gt_offset,train_grid,,,train_pt_fea) in enumerate(train_dataset_loader):
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in next
data = self._next_data()
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
return self._process_data(data)
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
data.reraise()
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/_utils.py", line 428, in reraise
raise self.exc_type(msg)
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
data = fetcher.fetch(index)
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/data/p202008021038/anaconda3/envs/ppnet/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/data/p202008021038/Panoptic-PolarNet-main/dataloader/dataset.py", line 313, in getitem
xyz,labels,insts,feat = self.inst_aug.instance_aug(xyz,labels.squeeze(),insts.squeeze(),feat)
File "/data/p202008021038/Panoptic-PolarNet-main/dataloader/instance_augmentation.py", line 40, in instance_aug
random_choice = np.random.choice(len(self.instance_path[self.thing_list[n]]),count)
File "mtrand.pyx", line 903, in numpy.random.mtrand.RandomState.choice
ValueError: a must be greater than 0 unless no samples are taken
0%| | 0/9565 [00:00<?, ?it/s]
进程已结束,退出代码为 1
I notice that visualize.py is included and I have installed open3d. But I didn't get the visualized result by running test_pretrain.py, neither for validation sequence and test sequence. Can you suggest how to get it?
need author's help, please tell how to fix it.
If anybody has tested this repo on Nuscenes panoptic segmentaion please ping me. i will be glad if one can help. Thank you in advance.
Thank you for your response @edwardzhou130. I am trying as you told me but getting this error when I try to run dataloader. I have changed lidarseg to panopticseg in line 124
.
Hello, i implement your code on SemanticKITTI dataset, and keep the default config setting.
However, i have cost 3 days to train the network, and the current epoch is 30. The max_epoch is 100. Following this setting, i will cost 10 days to train the whole pipeline.
The GPU i used is Tesla-V100 16G.
The process is too slow to train, so any suggestions?
Thanks sincerely~
I am getting issue with generating the Panoptic labels for nuscenes in test_pretrain_nuscenes. Please help me.
Hi, thank you for your interesting paper and releasing the code. I got a question about panoptic fusion on NuScenes. What NMS parameters for panoptic fusion you use for baseline PolarNet + CBGS in table 3? I use group nms following the multi-group head and nms parameters per group is
nms_pre_max_size=1000, nms_post_max_size=83, nms_iou_threshold=0.2, score_threshold=0.1
But I got panoptic quality as 31 instead of 67 for PolarNet + CBGS. I got decent SQ, but for RQ there are a lot of false positives for things.
I followed the evaluation code here https://github.com/edwardzhou130/Panoptic-PolarNet/blob/main/utils/eval_pq.py and change min_points to 20 as stated in your paper.
Good work. Are you planning to upload a dataset_nuscenes and train_nuscenes files? I really want to test it.
Thank you for your work!
I ran two experiments to reproduce the results of the pretrained model listed in repository:
Results for the pretrained model:
Validation per class PQ, SQ, RQ and IoU:
car : 87.94% 91.18% 96.45% 93.65%
bicycle : 52.57% 77.49% 67.84% 46.37%
motorcycle : 65.87% 88.72% 74.24% 66.71%
truck : 62.52% 87.94% 71.10% 88.46%
bus : 52.99% 88.41% 59.93% 43.35%
person : 73.23% 89.91% 81.45% 62.23%
bicyclist : 85.31% 90.66% 94.11% 82.21%
motorcyclist : 41.43% 84.35% 49.12% 48.48%
road : 93.74% 93.74% 100.00% 93.63%
parking : 24.17% 70.29% 34.38% 45.05%
sidewalk : 76.44% 83.27% 91.80% 80.26%
other-ground : 0.00% 0.00% 0.00% 0.48%
building : 87.46% 90.54% 96.60% 90.28%
fence : 18.05% 70.02% 25.77% 51.68%
vegetation : 84.88% 85.51% 99.26% 86.44%
trunk : 45.25% 72.43% 62.47% 61.96%
terrain : 56.98% 74.62% 76.36% 73.54%
pole : 55.89% 73.23% 76.32% 62.35%
traffic-sign : 54.20% 73.82% 73.43% 46.23%
Current val PQ is 58.890
Current val miou is 64.387
Results of my experiments:
Validation per class PQ, SQ, RQ and IoU:
car : 85.57% 89.42% 95.70% 92.69%
bicycle : 45.07% 77.62% 58.06% 38.38%
motorcycle : 60.85% 87.75% 69.34% 53.97%
truck : 66.72% 89.95% 74.17% 73.40%
bus : 52.96% 88.94% 59.55% 35.49%
person : 69.85% 89.19% 78.32% 56.55%
bicyclist : 87.17% 91.88% 94.87% 77.88%
motorcyclist : 41.59% 80.88% 51.43% 50.53%
road : 92.43% 92.45% 99.98% 92.30%
parking : 22.38% 68.82% 32.52% 39.85%
sidewalk : 74.24% 80.88% 91.80% 77.54%
other-ground : 0.00% 0.00% 0.00% 0.30%
building : 86.14% 90.27% 95.43% 88.60%
fence : 17.57% 67.88% 25.89% 48.37%
vegetation : 83.11% 85.61% 97.08% 85.21%
trunk : 35.16% 70.44% 49.91% 53.74%
terrain : 54.49% 73.21% 74.43% 69.84%
pole : 55.85% 73.17% 76.33% 62.68%
traffic-sign : 52.02% 75.09% 69.28% 43.30%
Current val PQ is 57.010
Current val miou is 60.034
Current val PQ is 55.956
Current val miou is 60.289
Both experiments were running for almost 100 epochs and their results (mIoU 60.3) don't match the results of your pretrained model (mIoU 64.3)
Could you please share the parameters and random seed for the experiment that achieved mIoU 64.3?
Thank you!
Hi. thank you for your last response. Now I am trying to deploy your model in one of my Jetson board with Velodyne sensor. First I want to try just feeding the single frame into the model. Can you suggest where can I change for input data in the test code. so I can easily generate predictions without giving data_path. My final goal is to have real-time segmentation with Lidar input. In more simplified way, input as a lidar raw data string and output as a predictions.
Hello, I am trying to run the code using command: python3 test_pretrain.py,
but return:
RuntimeError: Error(s) in loading state_dict for ptBEVnet:
size mismatch for BEV_model.network.inc.conv.0.weight: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.0.bias: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.0.running_mean: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.0.running_var: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.1.conv1.0.weight: copying a param with shape torch.Size([64, 32, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]).
in fact, I did not change the code of the paper.
Hi. I am trying to implement your code. I have input from Velodyne 64 as x,y,z,I. is it possible to use your model to directly use this values and obtain the predictions?
Thanks for your contribution and your work is very impressive and interesting. Just wondering about the check_occlusion
function in the part of instance augmentation ( https://github.com/edwardzhou130/Panoptic-PolarNet/blob/main/dataloader/instance_augmentation.py#L160 ). The goal of this function is to check whether a point is within a (2 metre) range of any point in a point cloud right? Would dist = np.linalg.norm(points-center,axis=0)
be dist = np.linalg.norm(points-center,axis=1)
instead? When you compute norm on axis 0, it will be a shape of (3, ) for [x, y, z], and on axis 1 will be a shape of (N, ) for number of points.
Congratulations on such a good work. I have tried to replicate the results as shown in the paper but after training for >50 epochs, I am still not getting the results similar to the paper. I have used the batch size of 3, and everything similar to the config file in the git repo.
In another issue, you have mentioned to use around 50 epochs and sap as 20.
Following are my results:
Validation per class PQ, SQ, RQ and IoU: car : 87.58% 91.26% 95.96% 93.75% bicycle : 27.15% 75.09% 36.16% 18.39% motorcycle : 54.03% 89.00% 60.71% 55.05% truck : 69.47% 90.71% 76.59% 85.95% bus : 48.70% 90.19% 54.00% 43.03% person : 69.89% 90.11% 77.55% 56.81% bicyclist : 87.22% 91.86% 94.95% 76.11% motorcyclist : 1.67% 68.28% 2.44% 0.97% road : 93.33% 93.35% 99.98% 93.31% parking : 25.02% 71.31% 35.08% 44.85% sidewalk : 75.14% 82.17% 91.45% 78.90% other-ground : 0.00% 0.00% 0.00% 0.37% building : 86.55% 90.47% 95.66% 88.56% fence : 14.98% 67.74% 22.11% 46.67% vegetation : 82.53% 85.78% 96.22% 84.80% trunk : 39.25% 71.35% 55.00% 55.20% terrain : 53.21% 73.15% 72.73% 68.09% pole : 53.14% 72.97% 72.82% 60.64% traffic-sign : 50.63% 75.39% 67.16% 39.80% Current val PQ is 53.657 while the best val PQ is 55.571 Current val miou is 57.435
Can you please guide on where the performance gap can be reduced. It is urgent for our current work and help will be really appreciated.
Thank you.
I want to do inference on some sequence. Since the space on my server is limited and I cannot even download the 80G velodyne data zip, so I copied labels and velodyne data of sequence 00 from other's server and put it in data/sequences/00.
I run 'python test_pretrained.py'. But the results all showed 0.00.
My velodyne dir is : /data/sequences/00/velodyne. The labels dir is: /data/sequences/00/labels.
Part of the returned results are the follows:
'''
0it [00:00, ?it/s]
Current val PQ is 0.000
Current val miou is 0.000
Inference time per 1 is nan seconds, postprocessing time is nan seconds per scan
'''
Does anyone know if I put the data wrongly or something?
Also, is it possible that I do inference on raw point cloud data?
Hi!
Thank you for your wonderful work!
Do you have plans on releasing the model pretrained on the NuScenes dataset?
Thank you!
Hello, I am trying to run the code using command: python3 test_pretrain.py,
but return:
RuntimeError: Error(s) in loading state_dict for ptBEVnet:
size mismatch for BEV_model.network.inc.conv.0.weight: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.0.bias: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.0.running_mean: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.0.running_var: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for BEV_model.network.inc.conv.1.conv1.0.weight: copying a param with shape torch.Size([64, 32, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]).
in fact, I did not change the code of the paper.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.