yzhuoning / awesome-clip

Awesome list for research on CLIP (Contrastive Language-Image Pre-Training).

Awesome CLIP

This repo collects the research resources based on CLIP (Contrastive Language-Image Pre-Training) proposed by OpenAI. If you would like to contribute, please open an issue.

CLIP

• Learning Transferable Visual Models From Natural Language Supervision [code]
• CLIP: Connecting Text and Images
• Multimodal Neurons in Artificial Neural Networks

Training

• OpenCLIP (3rd-party, PyTorch) [code]
• Train-CLIP (3rd-party, PyTorch) [code]
• Paddle-CLIP (3rd-party, PaddlePaddle) [code]

Applications

GAN

• VQGAN-CLIP [code]
• StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery [code]
• CLIP Guided Diffusion [code]
• CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions [code]
• TargetCLIP: Image-Based CLIP-Guided Essence Transfer [code]
• DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation [code]
• Clip2latent: Text driven sampling of a pre-trained StyleGAN using denoising diffusion and CLIP [code]

Object Detection

• Roboflow Zero-shot Object Tracking [code]
• Zero-Shot Detection via Vision and Language Knowledge Distillation [code]
• Crop-CLIP [code]
• Detic: Detecting Twenty-thousand Classes using Image-level Supervision [code]
• CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
• SLIP: Self-supervision meets Language-Image Pre-training [code]
• ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension [code]

Information Retrieval

• Unsplash Image Search [code]
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval [code]
• Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling [code]
• Natural Language YouTube Search [code]
• CLIP-as-service: Embed images and sentences into fixed-length vectors with CLIP [code]
• clip-retrieval [code]
• A CLIP-Hitchhiker’s Guide to Long Video Retrieval [code]
• CLIP2Video: Mastering Video-Text Retrieval via Image CLIP [code]
• X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval [code]
• Extending CLIP for Category-to-image Retrieval in E-commerce [code]

Representation Learning

• Wav2CLIP: Learning Robust Audio Representations From CLIP [code]
• CLIP-Lite: Information Efficient Visual Representation Learning from Textual Annotation [code]
• RegionCLIP: Region-based Language-Image Pretraining [code]
• CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classification [code]
• DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting [code]
• CyCLIP: Cyclic Contrastive Language-Image Pretraining [code]
• CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment [code]
• DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection [code]
• UniCLIP: Unified Framework for Contrastive Language–Image Pre-training [code]
• SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model [code]
• Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese [code]
• PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining [code]
• Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training [code]
• Fine-tuned CLIP Models are Efficient Video Learners[code]
• MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining [code]
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm [code]
• Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision [code]

Text-to-3D Generation

• CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation [code]
• Text2Mesh: Text-Driven Neural Stylization for Meshes [code]
• CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP [code]
• CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders [code]
• CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields [code]
• MotionCLIP: Exposing Human Motion Generation to CLIP Space [code]
• AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars [code]
• ClipFace: Text-guided Editing of Textured 3D Morphable Models [code]

Text-to-Image Generation

• Big Sleep: A simple command line tool for text to image generation [code]
• Deep Daze: A simple command line tool for text to image generation [code]
• CLIP-CLOP: CLIP-Guided Collage and Photomontage [code]
• CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP [code]

Prompt Learning

• Learning to Prompt for Vision-Language Models [code]
• Conditional Prompt Learning for Vision-Language Models [code]
• Prompt-aligned Gradient for Prompt Tuning [code]
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters [code]
• Learning to Compose Soft Prompts for Compositional Zero-Shot Learning [code]

Video Understanding

• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding [code]
• FitCLIP: Refining Large-Scale Pretrained Image-Text Models for Zero-Shot Video Understanding Tasks [code]
• Frozen CLIP Models are Efficient Video Learners [code]
• Towards Real-Time Text2Video via CLIP-Guided, Pixel-Level Optimization [code]
• MovieCLIP: Visual Scene Recognition in Movies [code]

Image Captioning

• CLIP prefix captioning [code]
• CLIPScore: A Reference-free Evaluation Metric for Image Captioning [code]
• ClipCap: CLIP Prefix for Image Captioning [code]
• Text-Only Training for Image Captioning using Noise-Injected CLIP [code]
• Fine-grained Image Captioning with CLIP Reward [code]

Image Editing

• HairCLIP: Design Your Hair by Text and Reference Image [code]
• CLIPstyler: Image Style Transfer with a Single Text Condition [code]
• CLIPasso: Semantically-Aware Object Sketching [code]
• Image-based CLIP-Guided Essence Transfer [code]
• CLIPDraw: Synthesize drawings to match a text prompt! [code]
• CLIP-CLOP: CLIP-Guided Collage and Photomontage [code]
• Towards Counterfactual Image Manipulation via CLIP [code]
• ClipCrop: Conditioned Cropping Driven by Vision-Language Model [code]
• CLIPascene: Scene Sketching with Different Types and Levels of Abstraction [code]

Image Segmentation

• CLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation [code]
• Image Segmentation Using Text and Image Prompts [code]
• Extract Free Dense Labels from CLIP [code]
• Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP [code]

3D Recognition

• PointCLIP: Point Cloud Understanding by CLIP [code]
• CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training [code]
• MotionCLIP: Exposing Human Motion Generation to CLIP Space [code]
• LidarCLIP or: How I Learned to Talk to Point Clouds[code]
• CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP [code]

Audio

• AudioCLIP: Extending CLIP to Image, Text and Audio [code]
• Wav2CLIP: Learning Robust Audio Representations from Clip [code]
• AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization [code]

Language Tasks

• CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment [code]

Object Navigation

• CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration [code]

Localization

• Adapting CLIP For Phrase Localization Without Further Training [code]

Others

• Multilingual-CLIP [code]
• CLIP (With Haiku + Jax!) [code]
• CLIP-Event: Connecting Text and Images with Event Structures [code]
• How Much Can CLIP Benefit Vision-and-Language Tasks? [code]
• CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning [code]
• CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory [code]
• CLIP-Event: Connecting Text and Images with Event Structures [code]
• CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracywith ViT-B and ViT-L on ImageNet [code]
• Task Residual for Tuning Vision-Language Models [code]

Acknowledgment

Inspired by Awesome Visual-Transformer.

awesome-clip's Issues

Please add the following papers

Text2Mesh : Their approach can modify a given mesh with given text/image information via CLIP text/image encoder.

Detecting Twenty-thousand Classes using Image-level Supervision : Which is a object detection research by facebook, they use CLIP text embedding as classifier weight.

The above papers are I want to add.

---

I think Crop-CLIP should be put at Object Detection

---

For more image manipuliation / generation applications, I summarized in my medium
You can add them if you think they are valueable.
Text-Driven Image Manipulation/Generation with CLIP

The list of above image is allocated at this google sheet

Adding CLIP-as-service?

CLIP-as-service is a low-latency high-scalability service for embedding images and text. It can be easily integrated as a microservice into neural search solutions.

Code: https://github.com/jina-ai/clip-as-service
Docs: https://clip-as-service.jina.ai/

Would be awesome to add to this awesome list! Thanks in advance!

（CVPR 2022）Please consider our weakly supervised semantic segmentation based on CLIP

CLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation
Paper: https://arxiv.org/abs/2203.02668
Code: https://github.com/CVI-SZU/CLIMS

Related work on prompt learning

Hi!

This is an excellent collection of CLIP-related works. We recently put out a preprint on prompt learning. It would be awesome if you could include our work under the Prompt Learning section. Below are the details:

• Learning to Compose Soft Prompts for Compositional Zero-Shot Learning [paper] [code]

Please let me know if you want me to send a PR instead.

Thank you!

wrong with code link of DetCLIP

In the Representation Learning part, the code link following DetCLIP is DeCLIP. I can't find the code of DetCLIP. Maybe it is close source.

About the usage of pseudolabels to enhance CLIP

This repository is very useful to learn about the works bootstrapping off CLIP, thank you for curating it!

We have just published on arXiv a work that investigates how to best use pseudolabels generated by CLIP to enhance CLIP itself. We believe this work to have good applicability for practitioners that want to adapt CLIP to novel tasks efficiently and with limited, or no, labeled data.

You can find the paper here and the code here

I'm happy to submit a pull request if needed :)

related work on information retrieval

Hello!

Thanks for creating this repository, it is super useful!

My colleagues and I recently finalized the work on using CLIP for information retrieval in e-commerce domain. The paper is called 'Extending CLIP for Category-to-image Retrieval in E-commerce', we presented it on ECIR 2022 a couple of month ago.
I would really appreciate it if you could add it to the Information Retrieval subsection. Here is the markdown code in case it is helpful:

• Extending CLIP for Category-to-image Retrieval in E-commerce [paper]

Please let me know if you'd rather me send a pull request.

Thank you!

The code of RegionCLIP is now public

Hi Zhuoning,

Thanks for your contribution for this nice repo!

Just wanted to give a update that our work RegionCLIP (CVPR 2022) is now public (https://github.com/microsoft/RegionCLIP). Feel free to give it a try!

PS: the name of RegionCLIP was misspelled.

Best,
Yiwu

Maybe add these papers related to few-shot image classification?

• Learning to Prompt for Vision-Language Models [paper][code]
• Conditional Prompt Learning for Vision-Language Models [paper][code]
• Prompt-aligned Gradient for Prompt Tuning [paper][code]
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters [paper][code]

New papers for image captioning

Hi!
Thanks for this great repository.
I'm searching for different papers that used CLIP for image captioning. I read image captioning papers in this repository but I think some papers can be added to this section:

• Distinctive Image Captioning via CLIP Guided Group Optimization
  paper link: link
• The Unreasonable Effectiveness of CLIP Features for Image Captioning:
  An Experimental Analysis
  paper link: link

Adding ICML 2023 paper

Thanks for creating this repository! It's a very comprehensive source of information.

Could you please add our ICML 2023 paper, POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained models
? The code is provided in this link?

We appreciate your help. Thanks!