New York University
Table of Contents
Visual Place Recognition exploded in popularity over the recent years. Visual Place Recognition (VPR) is often characterized as being able to recognize the same place despite significant changes in appearance and viewpoint. VPR is a key component of Spatial Artificial Intelligence, enabling robotic platforms and intelligent augmentation platforms such as augmented reality devices to perceive and understand the physical world. Visual place recognition is a challenging problem due to the vast range of ways in which the appearance of real-world places can vary.
Though the research has gained popularity in the previous years, the lack of dynamic data over multiple seasons of the same place and/or route for the purpose of VPR has made it significantly challenging to train and evaluate models for Long-term Visual Place Recognition. Even though oxford robocar provides this data, it contains only front view monocular vison, and depth camera. Other dataset over the seasons are either stationary or does not over dynamic weather or seasons.
This repo contains the code to generate such dynamic, multi seasonsonal data specicially for the purpose for Visual Place Recognition wither Autonomously or by manually driving the car. This code can Navigate all the default maps of carla simulator and collect multiple sensor data for training or evaluation of VPR models. The data is sequential hence this is suitable for semi-supervised, self-supervised, unsupervised models as well.
The objective of this project:
- Ability to collect VPR data (camera, IMU, GPS, semantic segmentation images) unmonitored in a remote server autonomously
- Develop large scale data collection pipeline for Visual Place Recognition
- Visualize the data during the data collection process
- Have manual control and autonmous modes implemented
- Multi camera support
- Dynamic weather and seasonal changes simulation
- Simple keyboard interface
This Project showed interesting results and we hope would help community better develop VPR models.
Use the README.md
to get started.
This project is built with the below given major frameworks and libraries. The code is primarily based on python. Some plugins (OpenCV and Matplotlib) are added for data visualization.
This section should list any major frameworks/libraries used to bootstrap your project. Leave any add-ons/plugins for the acknowledgements section. Here are a few examples.
Sample data is available at here
This project uses python. Hence python version 3.7.xx or higher should be installed. We recommend installing with Anaconda and adding it to the path of Windows. Git is also recommended to easily download files from GitHub. Alternatively you can also download the folder from GitHub directly. The links for both are given below
How to reproduce the result and or clone the repository
- Clone the repo
git clone https://github.com/sudharsan-007/Carla-vpr-data-collector.git
- Change Directory into the folder
cd Carla-vpr-data-collector
- Create conda environment
conda create -n carla-vpr-dc
- Download and Install Carla Simulator
- Install Carla Dependencies(change the version to your version of Carla Simulator)
pip3 install carla==0.9.4
- Install requirements
pip3 install requirements.txt
- Run main script
main.py
to spwan a vehicle and start driving it. PressR
to start collecting data.python3 main.py
Tests were conducted to verify the data collection using this method is as close to real world as possible. Various networks were trained and tested to verify that real world results match generated data.
Data collected using this repo has been tested in popular visual place recognition algorithms.
Model | Day/Night | Weather | No of params | recall rate (%) | triplet loss |
---|---|---|---|---|---|
ResNet18 + NetVLAD | Day | Clear-noon | 4.2M | 69.02% | 0.000342 |
ResNet18 + NetVLAD | Night | Clear | 4.2M | 47.43% | 0.000474 |
ResNet18 + NetVLAD | Day | Rain-sunset | 4.2M | 65.39% | 0.000377 |
ResNet50 + NetVLAD | Day | Rain-sunset | 18.65M | 72.92% | 0.00028 |
Distributed under the MIT License. See LICENSE.txt
for more information.
Sudharsan Ananth - [email protected]
Project Link: https://github.com/sudharsan-007/Carla-vpr-data-collector.git
Thanks to AI4CE Lab for fueling this research. We are grateful to Prof. Chen Feng for this excellent support and feedbacks through out this project. Also to Chao Chen for mentoring me through out this project.