shenxiang-vqa / mccav

MCCAV

Multimodal context-aware consistency alignment is crucial for enhancing the performance of visual language tasks. However, existing methods face challenges in integrating different levels of context-aware information and achieving alignment between context-aware image and language. To address these issues, this paper proposes a Multimodal Context-aware Consistency Alignment for Visual-language (MCCAV) tasks. Firstly, we construct three types of context-aware information in the language encoder to effectively integrate contextual semantics at different levels. Subsequently, positional information (relative/absolute) is embedded in the image encoder to build context-aware information with spatial relation reasoning. Finally, we design adaptive Jaccard similarity-based and manually parameterized Gaussian radial basis function-based algorithms for context-aware consistency alignment, effectively aligning language sequences containing context-aware information with image sequences having complex spatial relations for semantic similarity. To validate the generality and effectiveness of MCCAV, extensive qualitative and quantitative experiments are conducted on five benchmark datasets of Visual Question Answering (VQA) and Visual Grounding (VG). Experimental results demonstrate that our approach achieves state-of-the-art performance on multiple benchmark datasets.