yeonwoosung / pytorch_mixture-of-experts Goto Github PK

While MoE training typically uses a fixed capacity to distribute tokens evenly across all experts, my understanding is that inference involves activating experts based on predicted relevance via a softmax gate. However, your implementation seems to lack this differentiation between training and inference.

I try to change number of experts, but i find it dose not work well no matter what number experts i set.

For example, when n=10, the acc is 46% after 100 epochs. when n=3, the acc is 47% after 100 epochs. when n=1, the acc is 49% after 100 epochs.
So I want ask if the code is wrong?