lily11223344 / rosita

ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration

ROSITA

News & Updates

(24/08/2021)

• Release the demo to perform fine-grained semantic alignments using the pretrained ROSITA model.

(15/08/2021)

• Release the basic framework for ROSITA, including the pretrained base ROSITA model, as well as the scripts to run the fine-tuning and evaluation on three downstream tasks (i.e., VQA, REC, ITR) over six datasets.

Introduction

This repository contains source code necessary to reproduce the results presented in our ACM MM paper ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration, which encodes the cROSs- and InTrA-model prior knowledge in a in a unified scene graph to perform knowledge-guided vision-and-language pretraining. Compared with existing counterparts, ROSITA learns better fine-grained semantic alignments across different modalities, thus improving the capability of the pretrained model.

Performance

We compare ROSITA against existing state-of-the-art VLP methods on three downstream tasks. All methods use the base model of Transformer for a fair comparison. The trained checkpoints to reproduce these results are provided in finetune.md.

Tasks	VQA	REC			ITR
Datasets	VQAv2 dev | std	RefCOCO val | testA | testB	RefCOCO+ val | testA | testB	RefCOCOg val | test	IR-COCO R@1 | R@5 | R@10	TR-COCO R@1 | R@5 | R@10	IR-Flickr R@1 | R@5 | R@10	TR-Flickr R@1 | R@5 | R@10
ROSITA	73.91 | 73.97	84.79 | 87.99 | 78.28	76.06 | 82.01 | 67.40	78.23 | 78.25	54.40 | 80.92 | 88.60	71.26 | 91.62 | 95.58	74.08 | 92.44 | 96.08	88.90 | 98.10 | 99.30
SoTA-base	73.59 | 73.67	81.56 | 87.40 | 74.48	76.05 | 81.65 | 65.70	75.90 | 75.93	54.00 | 80.80 | 88.50	70.00 | 91.10 | 95.50	74.74 | 92.86 | 95.82	86.60 | 97.90 | 99.20

Installation

Software and Hardware Requirements

We recommand a workstation with 4 GPU (>= 24GB, e.g., RTX 3090 or V100), 120GB memory and 50GB free disk space. We strongly recommend to use a SSD drive to guarantee high-speed I/O. Also, you should first install some necessary package as follows:

• Python >= 3.6
• PyTorch >= 1.4 with Cuda >=10.2
• torchvision >= 0.5.0
• Cython

# git clone
$ git clone https://github.com/MILVLG/rosita.git 

# build essential utils
$ cd rosita/rosita/utils/rec
$ python setup.py build
$ cp build/lib*/bbox.cpython*.so .

Dataset Setup

To download the required datasets to run this project, please check datasets.md for details.

Pretraining

Please check pretrain.md for the details for ROSITA pretraining. We currently only provide the pretrained model to run finetuning on downstream tasks. The codes to run pretraining will be released later.

Finetuning

Please check finetune.md for the details for finetuning on downstream tasks. Scripts to run finetuning on downstream tasks are provided. Also, we provide trained models that can be directly evaluated to reproduce the results.

Demo

We provide the Jupyter notebook scripts for reproducing the visualization results shown in our paper.

Acknowledgment

We appreciate the well-known open-source projects such as LXMERT, UNITER, OSCAR, and Huggingface, which help us a lot when writing our codes.

Yuhao Cui (@cuiyuhao1996) and Tong-An Luo (@Zoroaster97) are the main contributors to this repository. Please kindly contact them if you find any issue.

Citations

Please consider citing this paper if you use the code:

@inProceedings{cui2021rosita,
  title={ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration},
  author={Cui, Yuhao and Yu, Zhou and Wang, Chunqi and Zhao, Zhongzhou and Zhang, Ji and Wang, Meng and Yu, Jun},
  booktitle={Proceedings of the 29th ACM International Conference on Multimedia},
  year={2021}
}