tom-roddick / mono-semantic-maps

Predicting Semantic Map Representations from Images with Pyramid Occupancy Networks

This is the code associated with the paper Predicting Semantic Map Representations from Images with Pyramid Occupancy Networks, published at CVPR 2020.

Data generation

In our work we report results on two large-scale autonomous driving datasets: NuScenes and Argoverse. The birds-eye-view ground truth labels we use to train and evaluate our networks are generated by combining map information provided by the two datasets with 3D bounding box annotations, which we rasterise to produce a set of one-hot binary labels. We also make use of LiDAR point clouds to infer regions of the birds-eye-view which are completely occluded by buildings or other objects.

NuScenes

To train our method on NuScenes you will first need to

1. Download the NuScenes dataset which can be found at https://www.nuscenes.org/download. Only the metadata, keyframe and lidar blobs are necessary.
2. Download the map expansion pack. Note that to replicate our original results you should use the original version of the expansion (v1.0). The later versions fixed some bugs with the original maps so we would expect even better performance!
3. Install the NuScenes devkit from https://github.com/nutonomy/nuscenes-devkit
4. Cd to mono-semantic-maps
5. Edit the configs/datasets/nuscenes.yml file, setting the dataroot and label_root entries to the location of the NuScenes dataset and the desired ground truth folder respectively.
6. Run our data generation script: python scripts/make_nuscenes_labels.py. Bewarned there's a lot of data so this will take a few hours to run!

Argoverse

To train on the Argoverse dataset:

1. Download the Argoverse tracking data from https://www.argoverse.org/data.html#tracking-link. Our models were trained on version 1.1, you will need to download the four training blobs, validation blob, and the HD map data.
2. Install the Argoverse devkit from https://github.com/argoai/argoverse-api
3. Cd to mono-semantic-maps
4. Edit the configs/datasets/argoverse.yml file, setting the dataroot and label_root entries to the location of the install Argoverse data and the desired ground truth folder respectively.
5. Run our data generation script: python scripts/make_argoverse_labels.py. This script will also take a while to run!

Training

Once ground truth labels have been generated, you can train our method by running the train.py script in the root directory:

python train.py --dataset nuscenes --model pyramid

The --dataset flag allows you to specify the dataset to train on, either 'argoverse' or 'nuscenes'. The model flag allows training of the proposed method 'pyramid', or one of the baseline methods ('vpn' or 'ved'). Additional command line options can be specified by passing a list of key-value pairs to the --options flag. The full list of configurable options can be found in the configs/defaults.yml file.

mono-semantic-maps's Issues

Can not find the function defination of render_bbox

Hello, the function 'render_bbox' in the utils of nuscenes seems to be undefined. Could you please support it?

KeyError when train argoverse dataset

Hi, thanks for the great work! I want to train this model on argoverse dataset, got following error:

=== Beginning epoch 1 of 200 ===
  0%|                                                                                                                                                | 0/50000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/media/new_data3/mono-semantic-maps-master/train.py", line 340, in <module>
    main()
  File "/media/new_data3/mono-semantic-maps-master/train.py", line 316, in main
    train(train_loader, model, criterion, optimiser, summary, config, epoch)
  File "/media/new_data3/mono-semantic-maps-master/train.py", line 34, in train
    for i, batch in enumerate(tqdm(dataloader)):
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/media/new_data3/mono-semantic-maps-master/src/data/augmentation.py", line 14, in __getitem__
    image, calib, labels, mask = self.dataset[index]
  File "/media/new_data3/mono-semantic-maps-master/src/data/argoverse/dataset.py", line 59, in __getitem__
    split, log, camera = self.examples[timestamp]
KeyError: 30314

It seems like examples are indexed by timestamp inside argoverse, which can not index by normal numbers. How can i fix this issue? Thanks.

My env is:

python 3.9.7
1.10.2+cu113

Any trained model?

Could you please provide any trained models to evaluate directly?

Temporal fusion code?

Hi!

Great work! Do you plan on releasing the code for the temporal fusion (described in Section 3.2) as well?

Cheers

FileNotFoundError: [Errno 2] No such file or directory:

torch\builder\windows\pytorch\aten\src\ATen\native\TensorShape.cpp:3550.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
==> Loading NuScenes dataset...

Loading NuScenes tables for version v1.0-mini...
23 category,
8 attribute,
4 visibility,
911 instance,
12 sensor,
120 calibrated_sensor,
31206 ego_pose,
8 log,
10 scene,
404 sample,
31206 sample_data,
18538 sample_annotation,
4 map,
Done loading in 0.476 seconds.

Reverse indexing ...
Done reverse indexing in 0.1 seconds.

=== Beginning epoch 1 of 100 ===
0%| | 0/125 [00:33<?, ?it/s]
Traceback (most recent call last):
File "\BEV\mono-semantic-maps-master\train.py", line 344, in
main()
File "\BEV\mono-semantic-maps-master\train.py", line 319, in main
train(train_loader, model, criterion, optimiser, summary, config, epoch)
File "\BEV\mono-semantic-maps-master\train.py", line 37, in train
for i, batch in enumerate(tqdm(dataloader)):
File "\Users\S\anaconda3\envs\torch\lib\site-packages\tqdm\std.py", line 1182, in iter
for obj in iterable:
File "\Users\S\anaconda3\envs\torch\lib\site-packages\torch\utils\data\dataloader.py", line 631, in next
data = self._next_data()
File "\Users\S\anaconda3\envs\torch\lib\site-packages\torch\utils\data\dataloader.py", line 1346, in _next_data
return self._process_data(data)
File "\Users\S\anaconda3\envs\torch\lib\site-packages\torch\utils\data\dataloader.py", line 1372, in _process_data
data.reraise()
File "\Users\S\anaconda3\envs\torch\lib\site-packages\torch_utils.py", line 722, in reraise
raise exception
FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "\Users\S\anaconda3\envs\torch\lib\site-packages\torch\utils\data_utils\worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "\Users\S\anaconda3\envs\torch\lib\site-packages\torch\utils\data_utils\fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "\Users\S\anaconda3\envs\torch\lib\site-packages\torch\utils\data_utils\fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "\BEV\mono-semantic-maps-master\src\data\augmentation.py", line 14, in getitem
image, calib, labels, mask = self.dataset[index]
File "\BEV\mono-semantic-maps-master\src\data\nuscenes\dataset.py", line 56, in getitem
image = self.load_image(token)
File "\BEV\mono-semantic-maps-master\src\data\nuscenes\dataset.py", line 66, in load_image
image = Image.open(self.nuscenes.get_sample_data_path(token))
File "\Users\S\anaconda3\envs\torch\lib\site-packages\PIL\Image.py", line 3247, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'C:\BEV\mono-semantic-maps-master\nuscenes\samples/CAM_BACK_LEFT/n015-2018-07-24-11-22-45+0800__CAM_BACK_LEFT__1532402938647423.jpg'

Please help me to resolve this issue, NOTE: I tried all the possible way to mention correct paths in config file , also when i try big dataset i am facing the memory error and when using the mini dataset i face this above mentioned error. kindly help me with this

Dear author, where is the "tempreal and sensor data fusion" part?

map_extents

map_extents: [-25., 1., 25., 50.]

Hello. I'm confuse about that why map _extents is not [-25., 0., 25., 50.]?

An error about training the nuScenes dataset

When I am training, I have the above error, how can I solve it?
Many thanks!

code & dataset release?

Hi Tom,

That's great work!
I would love to play around with the data and models :)

Do you plan to release anytime soon the following?

1. the dataset-creation pipeline from ArgoVerse and NuScenes
2. the created dataset (ground-truth maps etc.)
3. your networks and models.

I think all of the above would be of great value to the community.
Are you planning to do that soon? Or is it dependent on the paper first being published?

Thanks a lot!
Z.

Thank you very much for the code, but could you please explain how to test the model？？

Any plan for releasing data & code?

Is hflip really random?

Thanks very much for releasing the code!
It seems that you have used horizontal flip as an augmentation during training. But looking at this it seems that the images are not flipped at random but instead they are flipped all the time. Shouldn't it be flipped randomly?

How to demo or evaluation?

how can i visualize or make a demo for generated model? thanks.

Model overfitting and not learning

I used your script and generated the ground truth labels and then used your code to train the model. However, the model is only overfitting. Train IoUs increase consistently but validation IoUs do not increase.

generating labels

Hi, @anthonyhu I got some trouble in generating BEV semantic labels of nuScenes.
The label I generated is like:

Is it right? And how to visualize it in color? Waiting for your answer and thanks a lot!

When is the code available?

Earnestly waiting!

How long does it take to train the model?

Hello！Thanks for your good work ! could you please explain what GPU you used to train models, and how long does it take?

Code to remove any overlapping segments

Hi guys,

would it be possible to get the code to remove the overlapping segments?

Best.

Dear author,Exception: Error: You are using an outdated map version (%s)! Please go to https://www.nuscenes.org/download to download the latest map!

Exception: Error: You are using an outdated map version (%s)! Please go to https://www.nuscenes.org/download to download the latest map!

I would like to ask if it is not possible to use the v1.0 map expansion?

Thx

black region in generated labels

Thanks for your wonderful work and codes!

I used your scripts to generate the nuscens BEV labels, but it has some black regions in the drivable(green circle above), which have much bad effect in our performance evalutaion. I also have tried map-v1.0,-v1.2,-v1.3, and they all have the same problem.
Could you please give me any advice? Many thanks again!

any script for demo or evaluation?

i trained model in nuScenes datasets following the instruction.
how can i visualize or make a demo for generated model?