thunderbeee / zscl

Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models

WELCOM 👋

I'm Mingyuan MA

• 🔷🔶 I enjoy my undergrad study at UC Berkeley, majoring Statistics and Computer Science and graduating in May 2023
• 🔴⚪️ I am leaving from Cal to Harvard University to study Data Science in FALL 2023
• 🔭 I’m currently working on HPC lab at National University of Singapore (NUS) and BOBA lab at UC Berkeley
• 🌱 I’m currently doing reserach on Large Models Continual Learning and Human-AI Collaboration on Decision Making

zscl's Issues

ValueError: not enough values to unpack (expected 3, got 2)

Hello, thanks for your great work.

When I run the codes, I encounter the following errors. Are there some changes in the CLIP package ?
Can you provide any helpfully suggestions? Thanks a lot in advance.

File "/mnt/nas/share/home/lyk/code/ZSCL/mtil/src/models/finetune.py", line 14, in finetune model, train_preprocess, val_preprocess = clip.load(args.model, jit=False) ValueError: not enough values to unpack (expected 3, got 2)

cant find the “Validation_GCC-1.1.0-Validation_output.csv”

Thank you for open-sourcing the code about ZSCL, but I have a problem in reproducing the cil, I didn't find the corresponding Validation_GCC-1.1.0-Validation_output.csv in the code, I could only find Validation_GCC-1.1.0-Validation.tsv from Conceptual_Captions, but it doesn't work, can you put this csv file used in the code?

Text embeddings in distillation loss

In the distillation loss of continual-CLIP:

https://github.com/Thunderbeee/ZSCL/blob/main/cil/continual_clip/models.py#LL260C4-L260C4

Shouldn't you also do the opposite comparison too? Compare the current model embeddings of the ref_text with the original model embeddings of the ref_images.

Also, is the method is "LwF", shouldn't the logits_current be between the current model embeddings of the ref_images and the ref_texts, instead of being between the current model embeddings of the ref_images and the ref model embeddings of the ref_texts?

If I that isn't the case, there is no possibility of fine tuning the text encoder only. Why is this discarded for continuous CLIP?

Sorry if this questions are pretty basic.

About Target Datasets

Hi, congratulations on your excellent work.
I wonder how to download target datasets in the MTIL setting.
In the dataset.md, it seems imagenet-like datasets with distribution shifts are provided, rather than the datasets truly used.
Looking forward to your replay.

Runtime Error: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward

Hello, Im getting the error mentioned in the title. The explanation in torch.nn.functional.cross_entropy function says that we need to give either the class indexes of ground truth classes or class probabilities (which suits to this case since the ground truths are not strict 0 1 labels but predictions coming from the pretrained model). And the code implementation seems to be correct for the second case, but it gives me runtime error so I had to change it with ground truth class indexes which seem to be work well. I dont know if it would have a significant role to decrease the accuracy though. Any idea??

NotImplementedError while loading Distillation dataset

Hello, I try to use zscl method with the suggested datasets (ImageNet and Conceptual Captions). I prepared the distillation dataset using https://github.com/ml-jku/cloob repository so I have "Validation_GCC-1.1.0-Validation_output.csv" file already. But it throws me the following error:

0%| | 0/1301 [00:00<?, ?it/s]
Error executing job with overrides: []
Traceback (most recent call last):
File "main.py", line 51, in continual_clip
model.adaptation(task_id, cfg, train_dataset, train_classes_names)
File "/workspace/ZSCL/cil/continual_clip/models.py", line 48, in adaptation
self.train(task_id, cfg, train_dataset, train_classes_names)
File "/workspace/ZSCL/cil/continual_clip/models.py", line 230, in train
ref_images, ref_labels = next(ref_iter)
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataset.py", line 33, in getitem
raise NotImplementedError
NotImplementedError

I know that the problem is about Distillation Dataset (Conceptual Captions) but I cant fix it. Can anybody help??