google-research / snap

SNAP: Self-supervised Neural Maps for Visual Positioning and Semantic Understanding (NeurIPS 2023)

SNAP!
Self-Supervised Neural Maps
for Visual Positioning and Semantic Understanding

Paul-Edouard Sarlin · Eduard Trulls
Marc Pollefeys · Jan Hosang · Simon Lynen

NeurIPS 2023 Paper | Video | Poster

SNAP estimates 2D neural maps from multi-modal data like StreetView and aeral imagery.
Neural maps learn easily interpretable, high-level semantics through self-supervision alone
and can be used for geometric and semantic tasks.

This repository hosts the training and inference code for SNAP, a deep neural network that turns multi-modal imagery into rich 2D neural maps. SNAP was trained on a large dataset of 50M StreetView images with associated camera poses and aerial views. We do not release this dataset and the trained models, so this code is provided solely as a reference and cannot be used as is to reproduce any result of the paper.

Usage

The project requires Python >= 3.10 and is based on Jax and Scenic. All dependencies are listed in requirements.txt.

• The data is stored as TensorFlow dataset and loaded in snap/data/loader.py.
• Train SNAP with self-supervision:

python -m snap.train --config=snap/configs/train_localization.py \
    --config.batch_size=32 \
    --workdir=train_snap_sv+aerial

• Evaluate SNAP for visual positioning:

python -m snap.evaluate --config=snap/configs/eval_localization.py \
    --config.workdir=train_snap_sv+aerial \
    --workdir=.  # unused

• Fine-tune SNAP for semantic mapping:

python -m snap.train --config=snap/configs/train_semantics.py \
    --config.batch_size=32 \
    --config.model.bev_mapper.pretrained_path=train_snap_sv+aerial \
    --workdir=train_snap_sv+aerial_semantics

• Evaluate the semantic mapping:

python -m snap.evaluate --config=snap/configs/eval_semantics.py \
    --config.workdir=train_snap_sv+aerial_semantics \
    --workdir=.  # unused

BibTeX citation

If you use any ideas from the paper or code from this repo, please consider citing:

@inproceedings{sarlin2023snap,
  author    = {Paul-Edouard Sarlin and
               Eduard Trulls and
               Marc Pollefeys and
               Jan Hosang and
               Simon Lynen},
  title     = {{SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding}},
  booktitle = {NeurIPS},
  year      = {2023}
}

This is not an officially supported Google product.

snap's Issues

Some questions

Hello! I really enjoyed the object snap you shared on GitHub, and it provided me with many valuable insights. Here, I have some questions about the experimental data in the project.
When I looked at your code and documentation, I found no lab data attached to the project. Missing data can make it difficult to understand how the model works and evaluate its performance. Therefore, I am interested in the composition of your experimental data set and some of the sample data.
If it is convenient for you, could you share some details about the experimental data? For example, the structure of the dataset. Or, can you provide some sample of dataset so that I can better understand the structure and contents of the dataset? Thank you！

Dataset for evaluation

Hi, can you share the steps for setting up the evaluation dataset?
I don't know where to download the dataset and get the following error :

ValueError: Could not find any version for location osaka-n14_streetside_sceneviewpair_20views_trekkerquery_eval at ./datasets\lemming_streetview\osaka-n14_streetside_sceneviewpair_20views_trekkerquery_eval.