doc-doc / next-gqa

Can I Trust Your Answer? Visually Grounded Video Question Answering (CVPR'24, Highlight)

Can I Trust Your Answer? Visually Grounded Video Question Answering

Introduction

We study visually grounded VideoQA by forcing vision-language models (VLMs) to answer questions and simultaneously ground the relevant video moments as visual evidences. We show that this task is easy for human yet is extremely challenging for existing VLMs, revealing that the strong QA performance of these models may largely due to short-cut learning (e.g., language priors and spurious vision-text correlations) versus faithful multimodal reasoning. By defining grounded VQA, we hope to discourage such short-cut learning and spark more interpretable and trustworthy techniques. This repository holds our data and code to facilitate the study.

Environment

Assume you have installed Anaconda, please do the following to setup the environment:

>conda create -n videoqa python==3.8
>conda activate videoqa
>conda install pytorch==1.8.1 torchvision==0.9.1 cudatoolkit=11.1 -c pytorch -c nvidia 
>git clone https://github.com/doc-doc/NExT-GQA.git
>pip install -r requirements.txt

Preparation

Please create a data folder outside this repo, so you have two folders in your workspace 'workspace/data/' and 'workspace/NExT-GQA/'.

Please download the related video feature or raw videos. Extract the feature into workspace/data/nextqa/CLIPL/. If you download the raw videos, you need to decode each video at 6fps and then extract the frame feature of CLIP via the script provided in code/TempCLIP/tools/extract_feat.sh.

Please follow the instructions in code for training and testing the respective models.

Result Visualization (NExT-GQA)

Citation

@inproceedings{xiao2024can,
  title={Can i trust your answer? visually grounded video question answering},
  author={Xiao, Junbin and Yao, Angela and Li, Yicong and Chua, Tat-Seng},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={13204--13214},
  year={2024}
}

next-gqa's Issues

the caculateion of metric

The code in "eval_ground.py" includes three modes to calculate the score, I guess they are evaluations of gauss mask ground, post-hoc ground, and the merge of both. However, which metric do you adopt in the paper?

frame2time_val.json file and gsub_val.json file

Hello, how do I get the frame2time_val.json file and the gsub_val.json file in the nextqa dataset?

Absence of the enriched 10% of the positive questions.

Hello, you mentioned in your paper that "we enrich 10% of the positive questions by rephrasing each question (using GPT-4 [36]) with maximal 5 additional questions to form 𝑄+." I can't find this part of the code and the corresponding question data. Have you opened this part of the code and data?

If not, I sincerely hope that you can provide it so that I can better reproduce the results of your paper. I am willing to follow your work for further research.

Two train modules

In the main_dual.py, there are two train modules, which one should I use?

from train.trainval_gdqa import train, eval
from train.trainval_warm import train, eval

Difference between frame2time and upbd file

Hi there! Thanks for your great work.
What's the difference between frame2time_{split} and upbd_{split}? It looks exactly the same.

about QANUS

Hello, I saw the QANUS paper you published on arXiv in 2015, but the framework code link and Qa-sys example link in the paper are no longer valid. Do you still have your QANUS code?