Awesome Image Captioning![Awesome](https://camo.githubusercontent.com/64f8905651212a80869afbecbf0a9c52a5d1e70beab750dea40a994fa9a9f3c6/68747470733a2f2f617765736f6d652e72652f62616467652e737667)
A curated list of image captioning and related area. :-)
Contributing
Please feel free to send me pull requests or email ([email protected]) to add links. Markdown format:
- [Paper Name](link) - Author 1 et al, `Conference Year`. [[code]](link)
Change Log
- Mar. 06 Another related repository in medical domain updated
- Jun. 13 CVPR'20 and ACL'20 updated
Table of Contents
Papers
Survey
- A Comprehensive Survey of Deep Learning for Image Captioning - Hossain M et al,
arXiv preprint 2018
.
Before
- I2t: Image parsing to text description - Yao B Z et al,
P IEEE 2011
. - Im2Text: Describing Images Using 1 Million Captioned Photographs - Ordonez V et al,
NIPS 2011
. [project web] - Deep Captioning with Multimodal Recurrent Neural Networks - Mao J et al,
arXiv preprint 2014
.
2015
CVPR 2015
- Show and Tell: A Neural Image Caption Generator - Vinyals O et al,
CVPR 2015
. [code] [code] - Deep Visual-Semantic Alignments for Generating Image Descriptions - Karpathy A et al,
CVPR 2015
. [project web] [code] - Mind’s Eye: A Recurrent Visual Representation for Image Caption Generation - Chen X et al,
CVPR 2015
. - Long-term Recurrent Convolutional Networks for Visual Recognition and Description - Donahue J et al,
CVPR 2015
. [code] [project web]
ICCV 2015
- Guiding the Long-Short Term Memory Model for Image Caption Generation - Jia X et al,
ICCV 2015
. - Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images - Mao J et al,
ICCV 2015
. [code]
NIPS 2015
- Expressing an Image Stream with a Sequence of Natural Sentences - Park C C et al,
NIPS 2015
. [code]
ICML 2015
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention - Xu K et al,
ICML 2015
. [project] [code] [code]
arXiv preprint 2015
- Order-Embeddings of Images and Language - Vendrov I et al,
arXiv preprint 2015
. [code] - Generating Images from Captions with Attention - Mansimov E et al,
arXiv preprint 2015
. [code] - Learning FRAME Models Using CNN Filters for Knowledge Visualization - Lu Y, et al,
arXiv preprint 2015
. [code] - Aligning where to see and what to tell: image caption with region-based attention and scene factorization - Jin J et al,
arXiv preprint 2015
.
2016
CVPR 2016
- Image captioning with semantic attention - You Q et al,
CVPR 2016
. [code] - DenseCap: Fully Convolutional Localization Networks for Dense Captioning - Johnson J et al,
CVPR 2016
. [code] - What value do explicit high level concepts have in vision to language problems? - Wu Q et al,
CVPR 2016
. - Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data - Lisa Anne Hendricks et al,
CVPR 2016
. [code] - SPICE: Semantic Propositional Image Caption Evaluation - Anderson P et al,
ECCV 2016
. [code]
ACMMM 2016
- Image Captioning with Deep Bidirectional LSTMs - Wang C et al,
ACMMM 2016
. [code]
ACL 2016
- Multimodal Pivots for Image Caption Translation - Hitschler J et al,
ACL 2016
.
arXiv preprint 2016
- Image Caption Generation with Text-Conditional Semantic Attention - Zhou L et al,
arXiv preprint 2016
. [code] - DeepDiary: Automatic Caption Generation for Lifelogging Image Streams - Fan C et al,
arXiv preprint 2016
. - Learning to generalize to new compositions in image understanding - Atzmon Y et al,
arXiv preprint 2016
. - Generating captions without looking beyond objects - Heuer H et al,
arXiv preprint 2016
. - Bootstrap, Review, Decode: Using Out-of-Domain Textual Data to Improve Image Captioning - Chen W et al,
arXiv preprint 2016
. [code] - Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering - Liu H et al,
arXiv preprint 2016
. - Recurrent Highway Networks with Language CNN for Image Captioning - Gu J et al,
arXiv preprint 2016
.
2017
CVPR 2017
- Captioning Images with Diverse Objects - Venugopalan S et al,
CVPR 2017
. [code] - Top-down Visual Saliency Guided by Captions - Ramanishka V et al,
CVPR 2017
. [code] - Self-Critical Sequence Training for Image Captioning - Steven J et al,
CVPR 2017
. [code] - Dense Captioning with Joint Inference and Visual Context - Yang L et al,
CVPR 2017
. [code] - Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition - Yufei W et al,
CVPR 2017
. [code] - A Hierarchical Approach for Generating Descriptive Image Paragraphs - Krause J et al,
CVPR 2017
. [code] - Deep Reinforcement Learning-based Image Captioning with Embedding Reward - Ren Z et al,
CVPR 2017
. - Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects - Ting Y et al,
CVPR 2017
. - Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning - Lu J et al,
CVPR 2017
. [code] - Attend to You: Personalized Image Captioning with Context Sequence Memory Networks - CC Park et al,
CVPR 2017
. [code] - SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning - Chen L et al,
CVPR 2017
. [code] - Bidirectional Beam Search: Forward-Backward Inference in Neural Sequence Models for Fill-In-The-Blank Image Captioning - Qing S et al,
CVPR 2017
.
ICCV 2017
- Areas of Attention for Image Captioning - Pedersoli M et al,
ICCV 2017
. - Boosting Image Captioning with Attributes - Yao T et al,
ICCV 2017
. - An Empirical Study of Language CNN for Image Captioning - Gu J et al,
ICCV 2017
. - Improved Image Captioning via Policy Gradient Optimization of SPIDEr - Liu S et al,
ICCV 2017
. - Towards Diverse and Natural Image Descriptions via a Conditional GAN - Dai B et al,
ICCV 2017
. [code] - Paying Attention to Descriptions Generated by Image Captioning Models - Tavakoliy H R et al,
ICCV 2017
. - Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner - Chen T H et al,
ICCV 2017
. [code]
AAAI 2017
- Image Caption with Global-Local Attention - Li L et al,
AAAI 2017
. - Reference Based LSTM for Image Captioning - Chen M et al,
AAAI 2017
. - Attention Correctness in Neural Image Captioning - Liu C et al,
AAAI 2017
. - Text-guided Attention Model for Image Captioning - Mun J et al,
AAAI 2017
. [code]
NIPS 2017
- Contrastive Learning for Image Captioning - Dai B et al,
NIPS 2017
. [code]
TPAMI 2017
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge - Vinyals O et al,
TPAMI 2017
. [code]
arXiv preprint 2017
- MAT: A Multimodal Attentive Translator for Image Captioning - Liu C et al,
arXiv preprint 2017
. - Actor-Critic Sequence Training for Image Captioning - Zhang L et al,
arXiv preprint 2017
. - What is the Role of Recurrent Neural Networks (RNNs) in an Image Caption Generator? - Tanti M et al,
arXiv preprint 2017
. [code] - Self-Guiding Multimodal LSTM - when we do not have a perfect training dataset for image captioning - Xian Y et al,
arXiv preprint 2017
. - Phrase-based Image Captioning with Hierarchical LSTM Model - Tan Y H et al,
arXiv preprint 2017
. - Show-and-Fool: Crafting Adversarial Examples for Neural Image Captioning - Chen H et al,
arXiv preprint 2017
.
2018
CVPR 2018
- Neural Baby Talk - Lu J et al,
CVPR 2018
. [code] - Convolutional Image Captioning - Aneja J et al,
CVPR 2018
. - Learning to Evaluate Image Captioning - Cui Y et al,
CVPR 2018
. [code] - Discriminability Objective for Training Descriptive Captions - Luo R et al,
CVPR 2018
. [code] - SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text - Mathews A et al,
CVPR 2018
. - Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering - Anderson P et al,
CVPR 2018
. [code] - GroupCap: Group-Based Image Captioning With Structured Relevance and Diversity Constraints
- Chen F et al,
CVPR 2018
.
ECCV 2018
- Unpaired Image Captioning by Language Pivoting - Gu J et al,
ECCV 2018
. - Recurrent Fusion Network for Image Captioning - Jiang W et al,
ECCV 2018
. - Exploring Visual Relationship for Image Captioning - Yao T et al,
ECCV 2018
. - Rethinking the Form of Latent States in Image Captioning - Dai B et al,
ECCV 2018
. [code] - Boosted Attention: Leveraging Human Attention for Image Captioning - Chen S et al,
ECCV 2018
. - "Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention - Chen T et al,
ECCV 2018
.
AAAI 2018
- Learning to Guide Decoding for Image Captioning - Jiang W et al,
AAAI 2018
. - Stack-Captioning: Coarse-to-Fine Learning for Image Captioning - Gu J et al,
AAAI 2018
. [code] - Temporal-difference Learning with Sampling Baseline for Image Captioning - Chen H et al,
AAAI 2018
.
NeurIPS 2018
- Partially-Supervised Image Captioning - Anderson P et al,
NeurIPS 2018
. - A Neural Compositional Paradigm for Image Captioning - Dai B et al,
NeurIPS 2018
.
NAACL 2018
- Defoiling Foiled Image Captions - Wang J et al,
NAACL 2018
. - Punny Captions: Witty Wordplay in Image Descriptions - Chandrasekaran A et al,
NAACL 2018
. [code] - Object Counts! Bringing Explicit Detections Back into Image Captioning - Aneja J et al,
NAACL 2018
.
ACL 2018
- Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning - Sharma P et al,
ACL 2018
. [code] - Attacking visual language grounding with adversarial examples: A case study on neural image captioning - Chen H et al,
ACL 2018
. [code]
EMNLP 2018
- simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions - Liu et al,
EMNLP 2018
. [code]
arXiv preprint 2018
- Improved Image Captioning with Adversarial Semantic Alignment - Melnyk I et al,
arXiv preprint 2018
. - Improving Image Captioning with Conditional Generative Adversarial Nets - Chen C et al,
arXiv preprint 2018
. - CNN+CNN: Convolutional Decoders for Image Captioning - Wang Q et al,
arXiv preprint 2018
. - Diverse and Controllable Image Captioning with Part-of-Speech Guidance - Deshpande A et al,
arXiv preprint 2018
.
2019
CVPR 2019
- Unsupervised Image Captioning - Yang F et al,
CVPR 2019
. [code] - Engaging Image Captioning Via Personality - Shuster K et al,
CVPR 2019
. - Pointing Novel Objects in Image Captioning - Li Y et al,
CVPR 2019
. - Auto-Encoding Scene Graphs for Image Captioning - Yang X et al,
CVPR 2019
. - Context and Attribute Grounded Dense Captioning - Yin G et al,
CVPR 2019
. - Look Back and Predict Forward in Image Captioning - Qin Y et al,
CVPR 2019
. - Self-critical n-step Training for Image Captioning - Gao J et al,
CVPR 2019
. - Intention Oriented Image Captions with Guiding Objects - Zheng Y et al,
CVPR 2019
. - Describing like humans: on diversity in image captioning - Wang Q et al,
CVPR 2019
. - Adversarial Semantic Alignment for Improved Image Captions - Dognin P et al,
CVPR 2019
. - MSCap: Multi-Style Image Captioning With Unpaired Stylized Text - Gao L et al,
CVPR 2019
. - Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech - Aditya D et al,
CVPR 2019
. - Good News, Everyone! Context driven entity-aware captioning for news images - Biten A F et al,
CVPR 2019
. [code] - CapSal: Leveraging Captioning to Boost Semantics for Salient Object Detection - Zhang L et al,
CVPR 2019
. [code] - Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioning - Kim D et al,
CVPR 2019
. [code] - Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions - Cornia M et al,
CVPR 2019
. [code] - Exact Adversarial Attack to Image Captioning via Structured Output Learning With Latent Variables - Xu Y et al,
CVPR 2019
.
AAAI 2019
- Meta Learning for Image Captioning - Li N et al,
AAAI 2019
. - Learning Object Context for Dense Captioning - Li X et al,
AAAI 2019
. - Hierarchical Attention Network for Image Captioning - Wang W et al,
AAAI 2019
. - Deliberate Residual based Attention Network for Image Captioning - Gao L et al,
AAAI 2019
. - Improving Image Captioning with Conditional Generative Adversarial Nets - Chen C et al,
AAAI 2019
. - Connecting Language to Images: A Progressive Attention-Guided Network for Simultaneous Image Captioning and Language Grounding - Song L et al,
AAAI 2019
.
ACL 2019
- Dense Procedure Captioning in Narrated Instructional Videos - Shi B et al,
ACL 2019
. - Informative Image Captioning with External Sources of Information - Zhao S et al,
ACL 2019
. - Bridging by Word: Image Grounded Vocabulary Construction for Visual Captioning - Fan Z et al,
ACL 2019
.
BMVC 2019
- Image Captioning with Unseen Objects - Demirel et al,
BMVC 2019
. - Look and Modify: Modification Networks for Image Captioning - Sammani et al,
BMVC 2019
. [code] - Show, Infer and Tell: Contextual Inference for Creative Captioning - Khare et al,
BMVC 2019
. [code] - SC-RANK: Improving Convolutional Image Captioning with Self-Critical Learning and Ranking Metric-based Reward - Yan et al,
BMVC 2019
.
ICCV 2019
- Hierarchy Parsing for Image Captioning - Yao T et al,
ICCV 2019
. - Entangled Transformer for Image Captioning - Li G et al,
ICCV 2019
. - Attention on Attention for Image Captioning - Huang L et al,
ICCV 2019
. [code] - Reflective Decoding Network for Image Captioning - Ke L at al,
ICCV 2019
. - Learning to Collocate Neural Modules for Image Captioning - Yang X et al,
ICCV 2019
.
NeurIPS 2019
- Image Captioning: Transforming Objects into Words - Herdade S et al,
NeurIPS 2019
. - Adaptively Aligned Image Captioning via Adaptive Attention Time - Huang L et al,
NeurIPS 2019
. [code] - Variational Structured Semantic Inference for Diverse Image Captioning - Chen F et al,
NeurIPS 2019
. - Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations - Liu F et al,
NeurIPS 2019
. [code]
IJCAI 2019
- Image Captioning with Compositional Neural Module Networks - Tian J et al,
IJCAI 2019
. - Exploring and Distilling Cross-Modal Information for Image Captioning - Liu F et al,
IJCAI 2019
. - Swell-and-Shrink: Decomposing Image Captioning by Transformation and Summarization - Wang H et al,
IJCAI 2019
. - Hornet: a hierarchical offshoot recurrent network for improving person re-ID via image captioning - Yan S et al,
IJCAI 2019
.
EMNLP 2019
- Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach - Kim D J et al,
EMNLP 2019
. - TIGEr: Text-to-Image Grounding for Image Caption Evaluation - Jiang M et al,
EMNLP 2019
. - REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning - Jiang M et al,
EMNLP 2019
. - Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering - Changpinyo S et al,
EMNLP 2019
.
2020
AAAI 2020
- MemCap: Memorizing Style Knowledge for Image Captioning - Zhao et al,
AAAI 2020
. - Unified Vision-Language Pre-Training for Image Captioning and VQA - Zhou L et al,
AAAI 2020
. - Show, Recall, and Tell: Image Captioning with Recall Mechanism - Wang L et al,
AAAI 2020
. - Reinforcing an Image Caption Generator using Off-line Human Feedback - Hongsuck Seo P et al,
AAAI 2020
. - Interactive Dual Generative Adversarial Networks for Image Captioning - Liu et al,
AAAI 2020
. - Feature Deformation Meta-Networks in Image Captioning of Novel Objects - Cao et al,
AAAI 2020
. - Joint Commonsense and Relation Reasoning for Image and Video Captioning - Hou et al,
AAAI 2020
. - Learning Long- and Short-Term User Literal-Preference with Multimodal Hierarchical Transformer Network
for Personalized Image Caption - Zhang et al,
AAAI 2020
.
CVPR 2020
- Normalized and Geometry-Aware Self-Attention Network for Image Captioning - Guo L et al,
CVPR 2020
. - Object Relational Graph with Teacher-Recommended Learning for Video Captioning - Zhang Z et al,
CVPR 2020
. - Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs - Chen S et al,
CVPR 2020
.
ACL 2020
- Improving Image Captioning with Better Use of Caption - Shi Z et al,
ACL 2020
. - Cross-modal Coherence Modeling for Caption Generation - Alikhani M et al,
ACL 2020
. - Improving Image Captioning Evaluation by Considering Inter References Variance - Yi Y et al,
ACL 2020
. - MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning - Lei J et al,
ACL 2020
. - Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA - Kim H et al,
ACL 2020
.
Dataset
- nocaps, LANG:
English
- MS COCO, LANG:
English
. - Flickr 8k, LANG:
English
. - Flickr 30k, LANG:
English
. - AI Challenger, LANG:
Chinese
. - Visual Genome, LANG:
English
. - SBUCaptionedPhotoDataset, LANG:
English
. - IAPR TC-12, LANG:
English, German and Spanish
.
Image Captioning Challenge
Popular Implementations
PyTorch
TensorFlow
Torch
Others
Licenses
To the extent possible under law, Zhihong Chen has waived all copyright and related or neighboring rights to this work.