NLP Usage about g-mlp

[jaketae]

Hey @muhammadfhadli1453, thanks for filing this issue, and apologies for the belated reply.

I'll certainly try to add more documentation on model usage. For the time being, I'll try to explain more on this thread in the hopes of answering your question.

You've probably already seen this from the README:

>>> from g_mlp import gMLPForLanguageModeling
>>> model = gMLPForLanguageModeling()
>>> tokens = torch.randint(0, 10000, (8, 256))
>>> model(tokens).shape
torch.Size([8, 256, 256])

You can consider gMLPForLanguageModeling as a BERT-like encoder model. Since this is an encoder model, you would train this model via masked language modeling (MLM), where some parts of the tokens are replaced with [MASK] and the model has to predict the actual token for each mask token.

If anything is unclear, let me know. Thanks!

[jaketae]

Closing this for now, feel free to ping me if you have any questions!