rjk-git / bert2gpt2_for_machinetranslation

Use BERT as the encoder and GPT2 as the decoder for machine translation.

BERT2GPT2

描述

• 基于huggingface的Transformers库实现。在编码器端与解码器端分别借助BERT和GPT2预训练成果，以期待模型能够更好的热启动，在使用小规模平行语料进行机器翻译训练任务中能够提供帮助。同时借助GPT2的单向语言模型能力，使生成句子更加顺畅。

支持

• huggingface/models中支持的BERT模型和GPT2模型作为源语言，目标语言的编码器解码器。

使用

1. 准备训练文件和验证文件，每行为一对平行语料，默认以“\t”分隔。
2. 参考run_example.sh设置训练文件路径、模型保存路径、训练参数等。

bert2gpt2_for_machinetranslation's Issues

是否有训练资料？

请问作者是否还留有训练的资料？比如数据量、模型大小选择、训练结果、评分等？
非常感谢！

How to use low resourced language which is not avilabled for prtrained model?

1. We are working on NMT which involves a very low resourced language for which pretrained models for BERT or GPT2 are not available.How to deal with this situation?.
   2.If we work with e.g., EN- X (X is a low resourced language). What should we do?
   should we freeze and fine tune any of encoder or decoder.?

Need a better README

😄

some suggestions

Thank you very much for your translation project. I'm very benefited and have leaned more about it. Could you change it into a suitable one for various components？such as bert2bert2，gpt2gpt2，gpt2bert and so on. I believe it will be a great work of art

the program will hang over with Multiple graphics cards on one machine