weituo12321 / prevalent

large scale pretrain for navigation task

Prevalent: A Pretrained Generic VLN Agent

This repository contains source code to reproduce the results presented in the paper:

Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training, CVPR 2020
Weituo Hao*, Chunyuan Li*, Xiujun Li, Lawrence Carin, Jianfeng Gao

Pretrain

Our collected triplets can be downloaded here

The pretrained model can be downloaded here

R2R

• Please check here for experiment setup
• Please check here for PREVALENT application

CVDN

• Please check here for experiment setup
• Please check here for PREVALENT application

HANNA

• Please check here for experiment setup
• Please check here for PREVALENT application

Citation

If you use this code for your research, please cite our paper:

@article{hao2020prevalent,
  title={Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training},
  author={Hao, Weituo and Li, Chunyuan and Li, Xiujun and Carin, Lawrence and Gao, Jianfeng},
  journal={Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020}
}

prevalent's Issues

How to train and use it?

Hi,

Thank you for your work!

Could you please well documented to explain how to train and use it in the README.md? For example, how to train with HANNA, CVDN, R2R. And where is the pre-trained model?

Thank you.

Where is your main model?

Hi, Thank you for sharing nice work.
I interested in your work.
But, I can't find your main models.
Where is it?
And, could you please explain about your model training scripts?

question about visualizing the top-down map

Hi, thanks for sharing such a good job! And I'm trying to visualizing the top-down map by following your script in your 3D profile. But the map I got is a black image. Could you please tell what is the problem of it?

How to fine tune?

您好。请问是不是通过将预训练模型语言部分的hidden states输入到R2R-EnvDrop中作为WordEmbedding来fine tune的啊？

So, the whole thing presented here is a pretrained model without any source code ???

Where is the source code ? This repository contains only a pretrained model and 0 source code.

Where is the training script?

I appreciate this work so much, and can not wait to have a try. Could you please explain your model training scripts?