Shape and Texture biased pretrain models about shapetexturedebiasedtraining

[LiYingwei]

For the results on ImageNet, I use 8 Titan XP but it still takes a long period of time for training.

At the early stage of this project, I use ResNet-18 with a downsampled ImageNet to quickly see some signals.
The downsampled ImageNet in my initial experiments contains 200 classes (same as Tiny-ImageNet), 100,000 images (500 images per class) for training, and 10,000 images (50 images per class) for validation. These images are randomly selected from ImageNet.

[LiYingwei]

It's been a while since I ran the training script, so I am not that sure. The training time might also be different if the hardware (e.g., #CPU, #GPU) is different.

The training script prints a much accurate per-epoch time estimation, and then you could use that time to compute the total training time on your side. On my side, it takes about half a week for ResNet50, but, as again, it's been a while since I did experiments. If you want to speed up the training time, you can try to use DistributedDataParallel (instead of DataParallel) as the parallel strategy.

[LiYingwei]

As I mentioned in #1 , when you use a small training batch size, the learning rate should be accordingly small. Could you try to set train-batch as 256 or set lr as 0.025? I don't know if there are other issues to fix since the performance is too low. What dataset do you use for training?

My script was:

python imagenet.py -a resnet50 --data $DATA --epochs 100 --schedule 30 60 90 --gamma 0.1 --checkpoint $MODEL_DIR --gpu-id 0,1,2,3,4,5,6,7 -j 8 --train-batch 256 --lr 0.1 --num_classes 1000 --mixbn --style --alpha 0.5 --lr_schedule step --multi_grid --label-mix-alpha 1 --warm_lr 0. --warm 5

[LiYingwei]

Should be --label-gamma. Sorry for the confusion.

[LiYingwei]

Sure. I will check if I saved these models and get back to you. Please expect a little bit late response because I am catching a deadline next month.

[Chanfeechen]

Thank you very much! That will be very helpful : )

…

On Fri, Oct 23, 2020 at 13:04 Yingwei Li ***@***.***> wrote: Sure. I will check if I saved these models and get back to you. Please expect a little bit late response because I am catching a deadline next month. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALNH6PBFZQZRZGLSFVUWKBTSMHOU5ANCNFSM4S4ZXZ2Q> .

[Chanfeechen]

Hi Yingwei~ In order to better schedule for my experiments, would you mind telling me what hardware you use to training the models?

[Chanfeechen]

Thank you very much for the quick reply!!

[Chanfeechen]

It's me again :) Thank you for your patient and quick response! I wonder if you still remember how many days does it takes your to train a model (for ResNet50 or other models)? (A approximation will be just fine! So that I can have an estimate on my experiments.)

[Chanfeechen]

Thank you very much for your advice and reply. It's very helpful and important to my future research!! : )

[Chanfeechen]

Hi LiYing, I had tried to train the texture/shape biased model with label-gamma = 0,1.
( With training command:
python imagenet.py -a resnet50 --data $DATA --epochs 100 --schedule 30 60 90 --checkpoint $MODEL_DIR --gpu-id 0 --train-batch 64 --lr 0.1 --mixbn --style --multi_grid --labeled-gamma 1)
But I only got around 33% validate accuracy for the extreme biased models(labeled gamma =1/0). It's not even close to the reported number in paper Table 1 (which is around 76%). I wonder if I were using the same setting as the reported model in Table 1. If possible, would you mind to check the hyper parameter settings for that extreme models for me?
Or Could you spot on any possible issues that might cause this huge difference?
Thank you very much!

[Chanfeechen]

@LiYingwei I use ILSVRC2012-1k to train the model.
Thank you for your advise! I will try to set lr as 0.025.

[Chanfeechen]

@LiYingwei I can not find the parameter --label-mix-alpha in script imagenet.py. Could you please double check your training command? Or maybe we are using different version?

[Chanfeechen]

Gotha! Thank you! : )

[LiYingwei]

As promised, here are the Shape- and Texture-biased ResNet-50 models.

res50-shape-biased.pth.tar
res50-texture-biased.pth.tar

[Chanfeechen]

Thanks, I'll check them out. I really appreciated it!

…

On Tue, Nov 24, 2020 at 06:00 Yingwei Li ***@***.***> wrote: As promised, here are the Shape- and Texture-biased ResNet-50 models. res50-shape-biased.pth.tar <https://livejohnshopkins-my.sharepoint.com/:u:/g/personal/yli286_jh_edu/Ece6w9LyQhBIizrJ7V1Jg20BqPp733hDaaBHSgJE9oN2IA?e=RnCUH1> res50-texture-biased.pth.tar <https://livejohnshopkins-my.sharepoint.com/:u:/g/personal/yli286_jh_edu/ETI0wwE_YTxOl8rIaJRI3rcBry1BuAbRfxTAO1guLy5Fcw?e=w3uLMB> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALNH6PDV5SQSHV62P6RLTU3SRO37XANCNFSM4S4ZXZ2Q> .