luodian / generalizable-mixture-of-experts

GMoE could be the next backbone model for many kinds of generalization task.

Welcome to Generalizable Mixture-of-Experts for Domain Generalization

🔥 Our paper Sparse Mixture-of-Experts are Domain Generalizable Learners has officially been accepted as ICLR 2023 for Oral presentation.

🔥 GMoE-S/16 model currently ranks top place among multiple DG datasets without extra pre-training data. (Our GMoE-S/16 is initilized from DeiT-S/16, which was only pretrained on ImageNet-1K 2012)

Wondering why GMoEs have astonishing performance? 🤯 Let's investigate the generalization ability of model architecture itself and see the great potentials of Sparse Mixture-of-Experts (MoE) architecture.

Preparation

pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116

python3 -m pip uninstall tutel -y
python3 -m pip install --user --upgrade git+https://github.com/microsoft/tutel@main

pip3 install -r requirements.txt

Datasets

python3 -m domainbed.scripts.download \
       --data_dir=./domainbed/data

Environments

Environment details used in paper for the main experiments on Nvidia V100 GPU.

Environment:
	Python: 3.9.12
	PyTorch: 1.12.0+cu116
	Torchvision: 0.13.0+cu116
	CUDA: 11.6
	CUDNN: 8302
	NumPy: 1.19.5
	PIL: 9.2.0

Start Training

Train a model:

python3 -m domainbed.scripts.train\
       --data_dir=./domainbed/data/OfficeHome/\
       --algorithm GMOE\
       --dataset OfficeHome\
       --test_env 2

Hyper-params

We put hparams for each dataset into

./domainbed/hparams_registry.py

Basically, you just need to choose --algorithm and --dataset. The optimal hparams will be loaded accordingly.

License

This source code is released under the MIT license, included here.

Acknowledgement

The MoE module is built on Tutel MoE.

generalizable-mixture-of-experts's Issues

in Fig 2. head，Trunk and others

Hi，sir. How to determine whether the head and others. Are there annotations in the dataset?

Models not available

Hi,

Thanks for the wonderful research. Also, thanks for sharing your code in such a timely manner, it is very helpful.

I am accessing your code and was trying to access the models from https://entuedu-my.sharepoint.com/:f:/g/personal/libo0013_e_ntu_edu_sg/ElEEAWuckgZKlDsK6inCa0UBs-KJcCG49ef30wmFmmW1og?e=2Gl0w0

But I could not find the models. Can you please help with this?

Regards
Harkirat

FIg

Test command

Hi, your work is awesome. could you please provide the commands for test trained model?

Thanks

Unable to reproduce the domainbed accuracies

Hello!
I am facing trouble reproducing the accuracies reported in paper(https://arxiv.org/pdf/2206.04046v5.pdf). I used the default hyperparameters mentioned in the paper, and obtained the following results:

1. VLCS : 78.8(Obtained); 80.2(Reported in paper)
2. PACS : 86.9(Obtained); 88.1(Reported in paper)
3. OfficeHome : 72.7(Obtained); 74.2(Reported in paper)
   Can you please help me with this issue? Am I missing any implementation details?

non-MoE 的VIT训练效果比MoE-VIT效果还好？

论文中在Office home数据集上，gmoe比vit模型效果要好，可以分享non-MoE的VIT训练代码吗？

How to get CUB-DG dataset?

Hi, I really like this job! I am so interested in the CUB-DG dataset. How could I get it? Thanks!

Thank you for your reply! I have done some GMoE multi-seed experiments and often found that acc decreases as more seeds are added, could you teach me how to maintain a high acc even with multiple seeds? Or could you please tell me the specific muti-seed that you used in experiments?

Hi!
I'm sorry to bother you again. I have done some GMoE multi-seed experiments and often found that acc decreases as more seeds are added, could you please tell me the specific muti-seed that you used in official experiments?

Originally posted by @SiyuJi5 in #4 (comment)

TypeError: __init__() got an unexpected keyword argument 'default_cfg'

Hello! I want to train the SFMOE using terra incognita datatset. After downloading the dataset(using the given instructions) I tried running the train script using the following command:

python3 -m domainbed.scripts.train --data_dir=./domainbed/data/OfficeHome/ --algorithm SFMOE --dataset OfficeHome --test_env 2 

But I received the following error:

Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/sensei/projects/compositionality_analysis_remote/MoE-DG/domainbed/scripts/train.py", line 194, in algorithm = algorithm_class(dataset.input_shape, dataset.num_classes, File "/home/sensei/projects/compositionality_analysis_remote/MoE-DG/domainbed/algorithms.py", line 230, in init self.model = vision_transformer.deit_small_distilled_patch16_224(pretrained=True, num_classes=num_classes, moe_layers=['F'] * 5 + ['S'] + ['F'] * 5 + ['S'], mlp_ratio=4., num_experts=4, Hierachical=False).cuda() File "/home/sensei/projects/compositionality_analysis_remote/MoE-DG/./domainbed/vision_transformer.py", line 978, in deit_small_distilled_patch16_224 model = _create_vision_transformer( File "/home/sensei/projects/compositionality_analysis_remote/MoE-DG/./domainbed/vision_transformer.py", line 614, in _create_vision_transformer model = build_model_with_cfg( File "/home/sensei/projects/compositionality_analysis_remote/MoE-DG/comp_venv/lib/python3.8/site-packages/timm/models/helpers.py", line 531, in build_model_with_cfg model = model_cls(**kwargs) if model_cfg is None else model_cls(cfg=model_cfg, **kwargs) TypeError: init() got an unexpected keyword argument 'default_cfg'

Please help me resolve this.
Thanks!