Question about the noisy top-k gating about mixture-of-experts

[davidmrau]

Please have a look at the original paper. The idea is to approximate the hard top-k gating with noisy smoothing during training with the function noisy_top_k_gating. Hope this helps.

[MukundVarmaT]

Hi @davidmrau
can u please elaborate on what did u mean by your comment? I was not able to find how the gradients are backpropagated from the paper.

Thanks in advance

[davidmrau]

Sorry if I was not clear enough. For the load balancing, you are adding the noisy gating. The Top-k gating is indeed not differentiable are stated in the paper: While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A.