wisconsinaivision / rise

Domain Generalization through Distilling CLIP with Language Guidance

Domain Generalization through Distilling CLIP with Language Guidance

This repo is the official implementation of our ICCV 2023 paper "A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance".

Getting Started

Data Preparation

• Download PACS dataset from here
• Download VLCS dataset from here
• Download OfficeHome dataset from here
• Download Terra dataset from here

The dataset is structured as follows:

dataset
├── PACS
│   ├── Domain1
│   ├── Domain2
│   └── Domain3
│   └── Domain4
├── VLCS
│   ├── ...
├── OfficeHome
│   ├── ...
└── Terra
    ├── ...

Install

• Pytorch 1.7.1 (or later) from here
• CLIP from here
• Timm: pip install timm

Launch a sweep

python train_rise.py\
       --dataset "PACS" --seed 0 --output_folder "sweep1" --data_path "your datasets path"

The training record will be saved in the "results/output_folder".

# Train RISE with mix of teachers
CUDA_VISIBLE_DEVICES="0,1,..." python train_rise_mix_teacher.py\
       --dataset "PACS" --seed 0 --output_folder "sweep1" --data_path "your datasets path"

Training mix of teachers might need more than one GPU. Please adjust the GPU count as necessary.

View the results

python evaluate_results.py\
       --dataset "PACS" --output_folder "sweep1"

The model is selected by training-domain validation criteria.

Acknowledgments

The codebase is built upon OoD-Bench, JigenDG and DomainBed.

rise's Issues

A small spelling mistake and a question about the test result.

Hi, thanks for your good work, i've found a small error in https://github.com/OoDBag/RISE#view-the-results)

python evaluate_results.py\ --dataset "PACS" --output_foler "sweep1"

'output_foler' should be 'output_folder'

By the way, I have a question about the final result: In your paper, which result did you choose? The average corresponding test result or the average best test result? THANKS~

About the question of using CLIP to introduce more informativeness

Which resnet18 model should I use？

The CLIP weights pre-trained on Terra dataset?

Thanks for your impressive work!
For a fair comparison, I am wondering if it's possible to release the CLIP weights pre-trained on the Terra dataset?

Could i ask a few questions about the training result?

I have run the code in VLCS dataset successfully and got several test accuracy in each domain, but after that how can I get a final result in this dataset? By calculating the weighted average accuracy of these four different domains or some other methods like a simple average of those test accuracy? It seems the result higher a lot than that showed in paper(81.7%) when I calculated in the weighted average way, which makes me a little confused.

How do I load models from Hugging Face? I am encountering an error: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

excellent work! May I ask how many and what type of GPUs did you use for training?

About the source code

When will the source code be released?