prs-eth / pcaccumulation Goto Github PK

Hi, thanks for sharing this wonderful project. I am currently training the network by running

python main.py configs/waymo/waymo.yaml 4 1 --misc.mode=train --path.dataset_base_local=$YOUR_DATASET_FOLDER.

After 2-epoch training, the following errors occur:

CUDA out of memory. Tried to allocate 406.00 MiB (GPU O; 23.65 GiB total capacity; 21.61 GiB already allocated; 356.31 MiB free; 22.04 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF10%283/2859 [07:52<1:12:56, 1.70s/it]
CUDA out of memory. Tried to allocate 406.00 MiB (GPU 0; 23.65 GiB total capacity; 21.92 GiB already allocated; 96.31 MiB free; 22.29 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF10%. |284/2859 [07:53<1:01:19, 1.43s/it]
CUDA out of memory. Tried to allocate 406.00 MiB (GPU O; 23.65 GiB total capacity; 21.59 GiB already allocated; 346.31 MiB free; 22.05 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF10%| 284/2859 [07:54<1:11:43, 1.67s/it]
Traceback (most recent call last):
File "main.py", line 79, in
trainer.train()
File "/test/Flow-experiments/PCAccumulation-main/libs/trainer.py", line 258, in train
self. inference_one_epoch(epoch, 'train')
File, "/test/Flow-experiments/PCAccumulation-main/libs/trainer.py", line 243, in inference_one_epoch
self. update_tensorboard(stats_meter, curr_iter, phase)
File "/test/Flow-experiments/PCAccumulation-main/libs/trainer.py", line 119, in update_tensorboard
stats, message = compute_mean_iou_recall_precis ion(stats_meter['mos_metric'], self. mos_mapping)
TypeError: 'NoneType' object is not subscriptable。

It seems that all iterations are skipped due to being out of the CUDA memory. And I also set the batch size to 2 and the same problem occurs. Do you have any suggestions to solve this problem? (The training is on: Python 3.8.8, Pytorch 1.12.0+cu116, a NVIDIA TITAN RTX GPU)
Besides, I set the hyper-parameter ''iter_size'' from 1 to 2, and the problem seemed to be solved. But I am very worried about whether this would have a negative impact on the model training and could not get the experimental results in the paper. Would you share more details about the parameter settings for model training?

Thank you for your good work! Do you plan to share the training code? I want to train the model in my own dataset. Looking for your reply.

Hello,

i have noticed that used system memory increased over the course of the training. The training of the model than crashes when the system memory is completely filled up. It usually takes multiple Epochs. Have you noticed this issue before? Do you have an explanation, or some idea what could cause this issue and what i should look further into? I use the model with my own data, but have also noticed this issue when used with your provided Waymo data.

Thanks for your help in advance

Hi! Thanks for OA the brilliant work! Can I ask for a copy of visualization code of this work? Thx

Hi, thank you for the great project.

I would like to adapt other datasets to use with this Code but i am facing some issues in the GT generation.

Is there a timeline on when you plan to publish the preprocessing code or could you elaborate on the ground truth data format? Especially on what time indices represents the "current" frame and if the ego_motion_gt/bbox_tsfm represent the transformation between some previous and the current frame or between two adjacent frames.

Thanks in advance

Thanks a lot for your excellent work, but I have some difficulties downloading the processed dataset you kindly provided (due to the complex network reachability in my country... ToT). I wonder that do you have the plan to share the data processing code of Waymo/Nuscenes datasets? Thank u very much!

Hi. Thank you for your wonderful work. I am a student who is interested in your method.
I have three questions.

1. python main.py configs/nuscene/nuscene.yaml 10 1 --misc.mode=train --path.dataset_base_local=$MY_DATASET_FOLDER I trained the model like this code, what does 10 and 1 mean here?
2. Is it possible to train with datasets such as KITTI? If possible, how to preprocess datasets?
3. How to visualize the mapping results?

I look forward to hearing from you soon.

Thanks for sharing your great work!

I am trying to understand how the point cloud samples represents. Taking an example of a point cloud file from your nuscenes val set, it contains around 200k points, it actually represents 11 consecutive frames from your code, with each frame contains around 20k points.
Does it come from the ordering: (keyframe, sweep1, ..., sweep10) ? And what does the corresponding time indices (0, 1, ..., 10) mean? the unit is second? Is there any post-processing done for the fused point cloud ?

Thanks in advance!