rllab-snu / visual-graph-memory

Official GitHub Repository for paper "Visual Graph Memory with Unsupervised Representation for Visual Navigation", ICCV 2021

Visual-Graph-Memory

This is an official GitHub Repository for paper "Visual Graph Memory with Unsupervised Representation for Visual Navigation", which is accepted as a regular paper (poster) in ICCV 2021.

Setup

Requirements

The source code is developed and tested in the following setting.

• Python 3.6
• pytorch 1.4~1.7
• habitat-sim 0.1.7 (commit version: ee75ba5312fff02aa60c04f0ad0b357452fc2edc)
• habitat 0.1.7 (commit version: 34a4042c03d6596f1d614faa4891868ddaf81c04)

Please refer to habitat-sim and habitat-lab for installation.

To set the environment, run:

pip install -r requirements.txt

Habitat Data (Gibson, MP3D) Setup

Most of the scripts in this code build the environments assuming that the gibson/mp3d dataset is in habitat-lab/data/ folder.

The recommended folder structure of habitat-api (or habitat-lab):

habitat-api (or habitat-lab)
  └── data
      └── datasets
      │   └── pointnav
      │       └── gibson
      │           └── v1
      │               └── train
      │               └── val
      └── scene_datasets
          └── gibson_habitat
              └── *.glb, *.navmeshs  

otherwise, you should edit the data path in these lines.

VGM Demonstration

To visualize the VGM generation, run:

python vgm_demo.py --gpu 0 --num-proc 2

This command will show the online VGM generation during random exploration. The rendering window will show the generated VGM and the observations as follows:

Note that the top-down map and pose information are only used for visualization, not for the graph generation.

Imitation Learning

1. Data generation

   python collect_IL_data.py --ep-per-env 200 --num-procs 4 --split train --data-dir /path/to/save/data
   

   This will generate the data for imitation learning. You can find some examples of the collected data in IL_data folder, and look into them with show_IL_data.ipynb.
2. Training

   python train_bc.py --config configs/vgm.yaml --stop --gpu 0
   

3. Evaluation

Reinforcement Learning

The reinforcement learning code is highly based on habitat-lab/habitat_baselines. To train the agent with reinforcement learning (PPO), run:

python train_rl.py --config configs/vgm.yaml --version EXPERIMENT_NAME --diff hard --render --stop --gpu 0

Evaluation

We provide evaluation code and the pretrained model.

python evaluate_random.py --config configs/vgm.yaml --version-name test --eval-ckpt VGM_ILRL.pth --stop --diff hard

You can use "evaluate_dataset.py" to evaluate VGM on public image-goal nav dataset (https://github.com/facebookresearch/image-goal-nav-dataset)

git clone https://github.com/facebookresearch/image-goal-nav-dataset.git
python evaluate_dataset.py --config configs/vgm.yaml --version-name test --eval-ckpt VGM_ILRL.pth --stop --diff hard

In the above dataset, the provided pretrained model shows following performances.

Easy(SR)	Easy(SPL)	Medium(SR)	Medium(SPL)	Hard(SR)	Hard(SPL)	Overall(SR)	Overall(SPL)
0.76	0.40	0.76	0.56	0.62	0.49	0.71	0.48

Also, our VGM model shows following performances on NRNS Image-Goal Navigation dataset (https://meerahahn.github.io/nrns/data)

- Straight

Easy(SR)	Easy(SPL)	Medium(SR)	Medium(SPL)	Hard(SR)	Hard(SPL)	Overall(SR)	Overall(SPL)
0.81	0.54	0.82	0.70	0.67	0.54	0.77	0.60

- Curved

Easy(SR)	Easy(SPL)	Medium(SR)	Medium(SPL)	Hard(SR)	Hard(SPL)	Overall(SR)	Overall(SPL)
0.81	0.46	0.79	0.60	0.62	0.47	0.74	0.51

visual-graph-memory's Issues

How were the test episodes sampled?

Hi, @obin-hero. Sorry for imposing on you again, but I was still confused about the performance of VGM models. I went over your experimental settings and found that you "tested 1,007 sampled episodes for each difficulty level". I tried testing the pretrained model on the first 1007 episodes out of 1400, but the SR and SPL were lower. I was wondering how these episodes were sampled.

If you kindly offer detailed info about these test episodes, I can make a fair comparison.

Thanks a lot!

Can I get the source code used to evaluating baselines in your paper "VGM"?

@obin-hero I read your article "Visual Graph Memory with Unsupervised Representation for Visual Navigation" and found it inspiring. Last time you kindly answered my questions issued here, and your hints were helpful. I am now following your work and attempting to evaluate the baselines mentioned in your experiment section on my own test dataset.

I found that ANS, CNN + LSTM, SMT, Neural Planner and Exploration + SPTM were evaluated and compared in your paper. Some of these baselines were not designed for Habitat, so I found it hard to adapt their source code.

I was wondering whether you could share with me the re-implemented code used to assess these baselines in your experiments.

I will really appreciate it if you can provide me with some help.

My email is [email protected].

Thanks.

Some questions about training of VGM

@obin-hero Hello! I've got some issues when training VGM models from scratch using the config file "vgm.yaml".

I followed your training routines expounded in your 《Visual Graph Memory》supplementary, training a VGM model using imitation learning until exact overfitting, and then fine-tuning it using PPO for 10M frames.

The two phases of training are shown below:

(Note: the RL training was interrupted and resumed from the 1M-th frame, so the end is the 9M-th frame)

My issue is that my model at 10M-th frame underperforms the pre-trained model “VGM_ILRL.pth”. Their performances on the public image-goal nav dataset are compared below:

• VGM_ILRL.pth: SR=0.6207 | SPL=0.4943 | Avg step=81.5155
• My VGM_ILRL: SR=0.5750| SPL=0.4772| Avg step=73.5988

I wonder why there is so large a gap between the two models, given that the random seeds and RL training frames are unchanged.

Did you pick the model ("VGM_ILRL.pth") with the highest performance within a range centering around the 10M-th frame, or the very model at the 10M-th frame?

I really appreciate it if you provide me with some hints.

run error in collect_IL_data.py

==================================================
@obin-hero Hello! I've got some issues when I run python collect_IL_data.py .
My habitat version is v0.1.7 and habitat-sim version is v0.1.7.
I install them in the commit version mentioned in your README.md

There is an error:
"""
SPACE[000/014] STARTED Cantwell
/home/qiming/anaconda3/envs/vgm/lib/python3.6/site-packages/sklearn/base.py:315: UserWarning: Trying to unpickle estimator GaussianMixture from version 0.22.2.post1 when using version 0.24.2. This might lead to breaking code or invalid results. Use at your own risk.
UserWarning)
2023-04-24 21:02:40,059 initializing sim Sim-0
Process ForkServerProcess-1:
Traceback (most recent call last):
File "/home/qiming/Visual-Graph-Memory-master/env_utils/custom_habitat_sim.py", line 120, in create_sim_config
"angle"
File "/home/qiming/habitat-lab/habitat/sims/habitat_simulator/habitat_simulator.py", line 78, in overwrite_config
"""
NameError: position is not found on habitat_sim but is found on habitat_lab config.
It's also not in the list of keys to ignore: {'min_depth', 'hfov', 'max_depth', 'type', 'width', 'normalize_depth', 'height', 'angle'}
Did you make a typo in the config?
If not the version of Habitat Sim may not be compatible with Habitat Lab version: ANGLE: 0
HEIGHT: 64
HFOV: 30
ORIENTATION: [0, 2.6179938779914944, 0]
POSITION: [0, 0.88, 0]
TYPE: PanoramicPartRGBSensor
WIDTH: 21

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/qiming/anaconda3/envs/vgm/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/qiming/anaconda3/envs/vgm/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/qiming/anaconda3/envs/vgm/lib/python3.6/contextlib.py", line 52, in inner
return func(*args, **kwds)
File "/home/qiming/habitat-lab/habitat/core/vector_env.py", line 233, in _worker_env
env = env_fn(*env_fn_args)
File "/home/qiming/Visual-Graph-Memory-master/collect_IL_data.py", line 31, in make_env_fn
env = MultiSearchEnv(config=config_env)
File "/home/qiming/Visual-Graph-Memory-master/env_utils/task_search_env.py", line 91, in init
super().init(self._core_env_config, dataset)
File "/home/qiming/Visual-Graph-Memory-master/env_utils/custom_habitat_env.py", line 407, in init
self._env = Env(config)
File "/home/qiming/Visual-Graph-Memory-master/env_utils/custom_habitat_env.py", line 96, in init
id_sim=self._config.SIMULATOR.TYPE, config=self._config.SIMULATOR
File "/home/qiming/habitat-lab/habitat/sims/registration.py", line 19, in make_sim
return _sim(**kwargs)
File "/home/qiming/Visual-Graph-Memory-master/env_utils/custom_habitat_sim.py", line 66, in init
self.sim_config = self.create_sim_config(sensor_suites)
File "/home/qiming/Visual-Graph-Memory-master/env_utils/custom_habitat_sim.py", line 159, in create_sim_config
sim_sensor_cfg.parameters["hfov"] = str(sensor.config.HFOV)
AttributeError: 'habitat_sim._ext.habitat_sim_bindings.SensorSpec' object has no attribute 'parameters'
"""
Thanks a lot!

How do you generate PCL.pth

Can you share the code or steps like how to generate your PCL.pth?