liuhao-lh / smd

Pytorch implementation of 'Improving Self-supervised Lightweight Model Learning via Hard-aware Metric Distillation. In ECCV 2022'

Improving Self-supervised Lightweight Model Learning via Hard-aware Metric Distillation

A PyTorch implementation of our paper:

Improving Self-supervised Lightweight Model Learning via Hard-aware Metric Distillation (Video)

• Accepted at ECCV 2022 (Oral).

Self-supervised Distillation on ImageNet

Dependencies

If you don't have python 3 environment:

conda create -n SMD python=3.8
conda activate SMD

Then install the required packages:

pip install -r requirements.txt

Only multi-gpu, DistributedDataParallel training is supported; single-gpu or DataParallel training is not supported.

Get teacher network

To pre-train a unsupervised ResNet-50 model on ImageNet, run:

python main_simsiam.py \
  -a resnet50 \
  --dist-url 'tcp://localhost:10001' --multiprocessing-distributed --world-size 1 --rank 0 \
  --fix-pred-lr \
  [your imagenet-folder with train and val folders]

Or you can download the pre-trained teacher model from SimSiam.

Unsupervised Distillation

To distill a ResNet-18 model on ImageNet in an 4-gpu machine, run:

python main_distill.py \
  -a resnet18 \
  --dist-url 'tcp://localhost:10002' --multiprocessing-distributed --world-size 1 --rank 0 \
  --teacher_path [your pre-trained teacher model path] \
  [your imagenet-folder with train and val folders]

Linear Classification

With a pre-trained model, to train a supervised linear classifier on frozen features in an 4-gpu machine, run:

python main_lincls.py \
  -a resnet18 \
  --dist-url 'tcp://localhost:10003' --multiprocessing-distributed --world-size 1 --rank 0 \
  --pretrained [your checkpoint path]/checkpoint_0099.pth.tar \
  --lars \
  [your imagenet-folder with train and val folders]

The above command uses LARS optimizer and a default batch size of 4096.

Acknowledgement

This repository is partly built upon SimSiam, DisCo and SEED. Thanks for their great works!

Citation

If you use SMD in your research or wish to refer to the baseline results published in this paper, please use the following BibTeX entry.

@inproceedings{ECCV2022smd,
  title={Improving Self-supervised Lightweight Model Learning via Hard-Aware Metric Distillation},
  author={Liu, Hao and Ye, Mang},
  booktitle={European Conference on Computer Vision},
  pages={295--311},
  year={2022},
  organization={Springer}
}

smd's Issues

How to get the results in Table.5 and Table.6 in your paper?

Thanks for your great work! I want to reproduce the results in table 5 and 6, i.e. Statistical information for positive and negative mining. So could you tell me the implementation details or show me the code.

checkpoints

Thanks for the great work. Do you have any plans to release the checkpoints?

hyperparams for CIFAR100 dataset

Thanks for the great work! I tried to reproduce the results in table 1 and can't get the reported results. So I want to check whether the hyperparams is incorrectly used .

1. I used 2 rtx titan gpus and the hyperparams in paper and I noticed that the running scripts in README.md is on an 4-gpu machine, so you run the cifar-100 dataset on a 4-gpu machine with the reported hyperparams？
2. The common linear eval recipe is : a. lr : 30 with step decay [60, [80]] CMC or b. lr: 0.1 with cosine decay MoCo v2, but I noticed you set learning rate as 30 with cosine decay, is it a typo?
3. The result in table 1 is confusing. The teacher for column 1 and column 3 is resnet32x4. However the knn and linear acc is different. While the teacher for column 1 and column 4 is different, but the results is the same. Is it a typo too?