tsinghua-mars-lab / hdmapnet Goto Github PK

Hello,

I have some questions when reviewing the code base. I know the function of IPM Net in the model. I think the grid plane (BEV map) should transfer to the pixel level coordinate system based on the code logics. That's to say, the Ln 192 and 193 of homography.py is to multiply the external trins, intrins, and post_RTs to transform the grid plane.
However, the Ks (intrins) seems to be an identity matrix because the Ln 71 of the hdmapnet.py is empty. The Ks is not initialized by the intrins of the camera. It seems a strange error here.
By the way, why the post_RTs is None at the Ln 78 of the hdmapnet.py? It should be none or it is an error?

Hi,
it is March now. Any plan to release the complete project. I am looking forward to it.

感谢提供标签生成和评估脚本
1.Little bug in your devkit
data/rasterize.py file in line 22 you have defination:
def mask_for_lines(lines, mask, thickness, idx)

while in line 53 and 55 you switch the order of the last two argument:
mask_for_lines(new_single_line, map_mask, idx, thickness)

this is not correct and it will influence the data flow of metric evaluation enoumously

2.Some data label generating details
the visualization of the label generated based on your orginal script is:

the visualization of the label generated after my modification is:

which is actually more reasonable

3.Question about the papers:
(1)
I actually read your paper thoroughly in great detail and reimplement your decoder part into my architecture with some changes. But I keep your direction predict part. One confusion I have is that: In your paper you said that you use softmax as the activation function for classification and your gt label are all zero except the two opposite direction, which is 1. I am wondering what kind of loss function you use for direction prediction. Do you use the ordinary cross entropy even though you have two 1's? Or do you use binary cross entropy like what multilabel prediction would usually do? But if later, Then why not use sigmoid as the activation functions.

(2)
It can be seen from your script that your bev map is [-30, 30, 0.15] [-15,15,0.15]. Is this also the default settings in Table 2 in your paper? This is not clear in your paper. Also I am wondering the influence of x-y-range and resolution (in your script it is 0.15) on the metric

Thanks for the great work! I run the vis_labe.py following the readme. And I found that the gt lines projected on surrounding views had some misalignment with the real line elements. The results may be attributed to supposing the Z=0 for all map elements in https://github.com/Tsinghua-MARS-Lab/HDMapNet/blob/main/vis_label.py#L75-L76

Will the gt misalignment between top-down map and perspective images hinder the performance of the surrounding branch of HDMapNet? And how to align the gt?

'MultiLineString' object is not iterable

Great work for bird-eye view perception with multi-view images. I am wondering how long can it take before the source code is released

Hi, thank you for your complete and great code.
When i tried to use the command 'python evaluate.py' as readme.md said, I got some error message:
AttributeError: 'NoneType' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.
I have trained the model, but i can't solve the problem above.
Besides, there is another question: i could't find a argument named 'result_path' as your readme.md said. Should i add the argument by myself or the version of code has some different?
Thank you for your time.

i'm trying to use my own dataset to train hdmapnet, and my cameras are not horizontal（differ from nusenes)，so i want to know that is the plane generated from z_pitch_yaw better ? @simonyilunw @hangzhaomit @Mrmoore98 @liqi0126

HI, as mentioned in the title,
I'm not able to install the specified requirement.txt thus not able to run this codebase.
Could you provide a docker image for us to run your code?
I appreciate your effort a lot.

Thank you for your amazing work.

I notice that there is a resize operation in the data fetching phase that shrinks the image size.

However, to keep the projection relation between ground truth 3D lanes and the (resized)image pixels, we also need to adjust the camera intrinsic during the data fetching phase. (For example, if the image is downsampled by two, the first two rows of the intrinsic matrix should also become one half smaller)

In HDMapNet, because we use linear layers to learn the mapping between the front view and the BEV view, there is no need for intrinsic parameters, and the code works fine.

However, in IPMNet and Lift-Splat-Shoot, correct camera intrinsic parameters are essential to correctly project the FV features to BEV. The inconsistent camera intrinsic matrix could probably hurt the performance of IPMNet and LSS in the experiments mentioned in your paper. Did the ablation experiments in your paper also use the code in this repo to reproduce?

按我看代码的理解，在运行vis_pred.py时相当于是使用训练得到的model.pt对数据集进行推理以及可视化了吧？但是这里在读取数据集时使用的仍是data/dataset.py里的semantic_dataset函数，在这个python脚本里我注意到有读取nuscenes数据集的map和Lidar的数据。这个是用来做什么的呢？在推理阶段不可以直接输入六张图片得到地图分割的结果吗？
希望能解答一下

e.g.,
step1 train.py
step2 test.py
step3 evalute.py

Hey, guys, according to my test, the whole post-processing process takes 900ms, which will lead to low FPS?

Hi, nice work!
What kind of GPU do you used that produced the results in the paper?
Thanks

Hi, great work!

I wonder how to reproduce mAP of HDMapNet.
Can you give some codes or commands?

The mAP in HDMapNet paper is marked in yellow.

Thanks in advance!

Great work!
Just curious if the world coordinates are available...

Hi, thanks for your great work! I have a question regarding your architecture of the BEV decoder, according to the paper:

The BEV decoder is a fully convolutional network with 3 branches, namely semantic segmentation branch, instance embedding branch, and direction prediction branch…ResNet with three blocks is used as the BEV decoder.

If I understand it right, the input BEV feature to the decoder has shape , the segmentation branch has output shape (K' different for each branch). It's not very clear to me how a ResNet can be applied here.

My guess is you use a U-Net structure with a shared encoder(which uses the last 3 encoders of ResNet with dilated convs to avoid downsampling) and a separate decoder for each branch, but this seems unnecessary given that the input is already a feature vector. Or do you mean each decoder is simply made up of 3 residual blocks? Thanks!

Also as a minor request, could you upload icon/car.png so that we can run vis_label.py with the ego car icon included in the g.t. map label as well?

Hi, thx for your nice work! When I read the code and try ro run python vis_label.py to produce label, I meet this error:XIO: fatal IO error 0 (Success) on X server "localhost:10.0 after 18050 requests (18050 known processed) with 862 events remaining.
How can I solve this? I wonder if you could give some advice. Thanks for your time!

hi,
What is the resolution of the images sent to the PV image encoder?

thanks!

When will you release complete code/model?

Hi, @liqi17thu

I am wondering which set is applied in Table.1, validation set or mini-validation set? Thanks.

in which folder predicted result save?

In vectorize process, what dose the sort_points_by_dist function do?

Hello I have a Data set, how to predict my data and detect its lane and how to visualize it?
if we want to get visualize all of the sample in nuScens what's should I do?

Hi, @liqi17thu @hangzhaomit ,

When will the code be released? The link (https://github.com/Tsinghua-MARS-Lab/HDMapNet-dev) attached in your website is broken now.

Thanks~

Hi, first thanks for your great job!
I wonder whether you will release your trained weights? thanks.

I just found that --direction-pred only works when --instance-seg is activated

because direction_pred and instance_seg share the self.up1_embedded layer

i guessed it is kinda typo. because there is another layer designed for direction_pred, 'self.up1_direction'. but never used

I saw a temporal fusion in the paper,this part of the code is not open source?

Hello! This is a great work! I am a new commer in this field, and I am confused with the threshold setting of CD in evaluate_json.py, where you set THRESHOLDS = [2, 4, 6], which is different from the [0.2, 0.5, 1] that is claimed in your paper. Will this cause different evaluation result of AP? I would appreciate it a lot if someone can explain it to me.

I am trying to use vis_label.py with nuScenes v1.0 mini dataset but it is not working.
This is the error
Traceback (most recent call last): File "/content/HDMapNet/vis_label.py", line 95, in <module> vis_label(args.dataroot, args.version, args.xbound, args.ybound) File "/content/HDMapNet/vis_label.py", line 24, in vis_label dataset = HDMapNetDataset(version=version, dataroot=dataroot, data_conf=data_conf, is_train=False) File "/content/HDMapNet/data/dataset.py", line 31, in __init__ self.nusc = NuScenes(version=version, dataroot=dataroot, verbose=False) File "/usr/local/lib/python3.10/dist-packages/nuscenes/nuscenes.py", line 62, in __init__ assert osp.exists(self.table_root), 'Database version not found: {}'.format(self.table_root) AssertionError: Database version not found: dataroot=/content/HDMapNet/dataset/nuScenes/v1.0-mini

I am working with NVIDIA Orion AGX production board and I have jetpack 5 installed on it which has cuda 11.4 installed. Every other libraries installed smoothly except these ones (torch==1.10.2+cu113 torchaudio==0.10.2+cu113 torchvision==0.11.3+cu113). Kindly help me with this issue.

I'm looking forword to the code. thanks.

How to figure this out, thx!

TRAIN[ 0]: [ 449/7031] Time: 0.4546 Loss: 1.3768 IOU: [0.000 0.000 0.000]

this is log of my training, IOU is always 0.0
is that right?

To create submission file try to run -

python export_gt_to_json.py

After running above file you will get submission.json file.

To evaluate file

python evaluate_json.py --result_path [submssion_file_path]

Hope this will save time for those who getting stuck with Readme instructions

hi,
Is the evaluation result in Table 1 of the paper on the nuscene val set or the mini-val set?
thanks!!!

Hi, thx for your nice work! When I train the model, I find a problem. In train dataset, IOU can achieve about[0.8 0.6 0.8], while its eval IOU is only about [0.04 0.01 0.05]. Here it is:

So Is there any wrong with function eval_iou()? It seems to be an order of magnitude smaller. Thanks for your time.

Thanks for your great research. Will you public complete project？

It seems that the arguments do not match for LiftSplat. Do you carry out experiments of LiftSplat on this codebase?

Nice work! I'm sure the whole code, including dataloader, model and evaluation, will provide the community a great guide to this task.

I wanted to ask if I could get the pre-trained weights of HDMapNet models that produced the results in the paper.

I would really appreciate it.

Thank you very much

Hello, I am really amazed by your work, HDMapNet shows great performance.
But apart from HDMapNet, the other networks in your code including VPN, Lift splat and IPMNet all have bugs in it and are unable to run, which really harm the credibility of this work and is really regretful.
I do believe that maybe you have others version to successfully run VPN, Lift splat and IPMNet but has not yet update them to your HDMapNet pipeline, could you please provide the correct code of them? Thank you very much!

Hi, first thanks for your great job!
I wonder whether you will release your training code? thanks.