Vision-and-language navigation (VLN) enables the agent to navigate to a remote location following the natural language instruction in 3D environments. At each navigation step, the agent selects from possible candidate locations and then makes the move. For better navigation planning, the lookahead exploration strategy aims to effectively evaluate the agent's next action by accurately anticipating the future environment of candidate locations. To this end, some existing works predict RGB images for future environments, while this strategy suffers from image distortion and high computational cost. To address these issues, we propose the pre-trained hierarchical neural radiance representation model (HNR) to produce multi-level semantic features for future environments, which are more robust and efficient than pixel-wise RGB reconstruction. Furthermore, with the predicted future environmental representations, our lookahead VLN model is able to construct the navigable future path tree and select the optimal path branch via efficient parallel evaluation. Extensive experiments on the VLN-CE datasets confirm the effectiveness of our method.
- Release the pre-training code of the Hierarchical Neural Radiance Representation Model.
- Release the checkpoints of the Hierarchical Neural Radiance Representation Model.
- Tidy the pre-training code for easy execution.
- Release the fine-tuning code of the Lookahead VLN Model.
- Release the checkpoints of the Lookahead VLN Model.
For training speed, see Issue#7
Load only a few scenes for efficient debugging, see Issue#4
- Install
Habitat simulator: follow instructions from ETPNav and VLN-CE. - Download the
Habitat-Matterport 3D Research Dataset (HM3D)from habitat-matterport-3dresearchhm3d-train-habitat-v0.2.tar hm3d-val-habitat-v0.2.tar - Download annotations (PointNav, VLN-CE) and trained models from Baidu Netdisk or TeraBox.
- Download pre-trained
waypoint predictorfrom link. - Install
torch_kdtreefor K-nearest feature search from torch_kdtree.git clone https://github.com/thomgrand/torch_kdtree cd torch_kdtree git submodule init git submodule update pip3 install . - Install
tinycudannfor faster multi-layer perceptrons (MLPs) from tiny-cuda-nn.pip3 install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
bash run_r2r/nerf.bash train 2345
Evaluate the cosine similarity between the HNR model's predicted features and the CLIP model's ground truth features.
bash run_r2r/nerf.bash eval 2345
Set Visualization to True in line 68 of HNR-VLN/NeRF/ss_trainer_ETP.py, visualize and save the images predicted by the HNR model.
@InProceedings{Wang_lookahead,
author = {Wang, Zihan and Li, Xiangyang and Yang, Jiahao and Liu, Yeqi and Hu, Junjie and Jiang, Ming and Jiang, Shuqiang},
title = {Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2024},
pages = {13753-13762}
}Our code is based on ETPNav, nerf-pytorch and torch_kdtree. Thanks for their great works!
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent