yeonwoosung / pytorch_mixture-of-experts

PyTorch implementation of moe, which stands for mixture of experts

The Sparsely Gated Mixture of Experts Layer for PyTorch

This repository contains the PyTorch re-implementation of the MoE layer described in the paper Outrageously Large Neural Networks for PyTorch.

Requirements

This example was tested using torch v1.0.0 and Python v3.6.1 on CPU.

To install the requirements run:

pip install -r requirements.txt

Example

The file test.py contains an example illustrating how to train and evaluate the MoE layer with dummy inputs and targets. To run the example:

python test.py

Acknowledgements

The code is based on the TensorFlow implementation that can be found here.

pytorch_mixture-of-experts's Issues

Do training and inference of MoE share the same dispatching method?

While MoE training typically uses a fixed capacity to distribute tokens evenly across all experts, my understanding is that inference involves activating experts based on predicted relevance via a softmax gate. However, your implementation seems to lack this differentiation between training and inference.

This MoE is not useful.

I try to change number of experts, but i find it dose not work well no matter what number experts i set.

For example, when n=10, the acc is 46% after 100 epochs. when n=3, the acc is 47% after 100 epochs. when n=1, the acc is 49% after 100 epochs.
So I want ask if the code is wrong?