Experiments and reproduction of pretrained models trained on WMT17 Chinese-English using fairseq
A big pain point for any RNN/LSTM model training is that they are very time consuming, so fairseq proposed fully convolutional architecture is very appealing. Some cursory experiments show much faster training time for fconv (Fully Convolutional Sequence-to-Sequence) compared to blstm (Bi-LSTM), while yielding comparable results. While fconv measures slightly worse BLEU scores vs blstm, some manual tests seem to favor fconv. A hybrid model using convenc (Convolutional encoder, LSTM decoder) trains for much more epochs but performs much worse BLEU score.
| Model | Epochs | Training Time | BLEU4 (beam1) | BLEU4 (beam5) | BLEU4 (beam10) | BLEU4 (beam20) |
|---|---|---|---|---|---|---|
| fconv | 25 | ~4.5hrs | 63.49 | 62.22 | 62.52 | 62.74 |
| fconv_enc7 | 33 | ~5hrs | 66.40 | 65.52 | 65.8 | 65.96 |
| fconv_dec5 | 28 | ~5hrs | 65.65 | 64.71 | 64.91 | 64.98 |
| blstm | 30 | ~8hrs | 64.59 | 64.15 | 64.38 | 63.76 |
| convenc | 47 | ~7hrs | 50.91 | 56.71 | 56.83 | 53.66 |
Pretrained models:
- wmt17.zh-en.fconv-cuda: Pre-trained model for WMT17 Chinese-English
- wmt17.zh-en.fconv-float: CPU version of the above
Follow fairseq installation, then:
# Chinese tokenizer
$ pip install jieba
# English tokenizer
$ pip install nltk
$ mkdir -p ~/nltk_data/tokenizers/
$ wget https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/packages/tokenizers/punkt.zip -o ~/nltk_data/tokenizers/punkt.zip
$ unzip ~/nltk_data/tokenizers/punkt.zip ~/nltk_data/tokenizers/
Additionally, we use scripts from Moses and Subword-nmt
git clone https://github.com/moses-smt/mosesdecoder
git clone https://github.com/rsennrich/subword-nmt
CUDA might need to link libraries to path.
# Couldn't open CUDA library libcupti.so.8.0. LD_LIBRARY_PATH: /git/torch/install/lib:
$ cd $LD_LIBRARY_PATH;
$ sudo ln -s /usr/local/cuda-8.0/extras/CUPTI/lib64/* $LD_LIBRARY_PATH/
We tokenize dataset, using nltk.word_tokenizer for English and jieba for Chinese word segmentation.
We remove cases from English and converted all string to lower case.
We note that dataset often has blank lines. In some cases, this is formatting, but there are cases where a long English sentence is translated to 2 Chinese sentences. This appears as a sentence followed by blank line on English corpus. To deal with this, we merge the 2 Chinese sentences onto same line, and then remove the blank line from both corpus.
There are also formatting issues, where English corpus has blank line while Chinese corpus has a single .. We treat this as both blank lines and remove them.
We note that there are further work that can be added to data cleaning:
- remove non-English/Chinese sentences. I think there was a Russian sentence.
- remove (HTML?) markup
- remove non-breaking white space.
\xa20was found and converted to whitespace.
Preprocessing is run by wmt_prepare.sh.
-
We download, unzip, tokenize, and clean dataset in
preprocess/wmt.py. -
We learn subword vocabulary using apply_bpe.
-
Then preprocess datasets to binary using
fairseq preprocess
Run wmt17_train.sh which does the following:
$ DATADIR=data-bin/wmt17_en_zh
$ TRAIN=trainings/wmt17_en_zh
# Standard bi-directional LSTM model
$ mkdir -p $TRAIN/blstm
$ fairseq train -sourcelang en -targetlang zh -datadir $DATADIR \
-model blstm -nhid 512 -dropout 0.2 -dropout_hid 0 -optim adam -lr 0.0003125 \
-savedir $TRAIN/blstm
# Fully convolutional sequence-to-sequence model
$ mkdir -p $TRAIN/fconv
$ fairseq train -sourcelang en -targetlang zh -datadir $DATADIR \
-model fconv -nenclayer 4 -nlayer 3 -dropout 0.2 -optim nag -lr 0.25 -clip 0.1 \
-momentum 0.99 -timeavg -bptt 0 -savedir $TRAIN/fconv
# Convolutional encoder, LSTM decoder
$ mkdir -p trainings/convenc
$ fairseq train -sourcelang en -targetlang zh -datadir $DATADIR \
-model conv -nenclayer 6 -dropout 0.2 -dropout_hid 0 -savedir trainings/convenc
Run wmt17_generate.sh, or run generate-line as follows:
$ DATADIR=data-bin/wmt17_en_zh
# Optional: optimize for generation speed (fconv only)
$ fairseq optimize-fconv -input_model trainings/fconv/model_best.th7 -output_model trainings/fconv/model_best_opt.th7
$ fairseq generate-lines -sourcedict $DATADIR/dict.en.th7 -targetdict $DATADIR/dict.zh.th7 -path trainings/fconv/model_best_opt.th7 -beam 10 -nbest 2
# you actually have to implement the solution
# <unk> 实际上 必须 实施 解决办法 。
$ fairseq generate-lines -sourcedict $DATADIR/dict.en.th7 -targetdict $DATADIR/dict.zh.th7 -path trainings/blstm/model_best.th7 -beam 10 -nbest 2
# you actually have to implement the solution
# <unk> ， 这些 方案 必须 非常 困难
$ fairseq generate-lines -sourcedict $DATADIR/dict.en.th7 -targetdict $DATADIR/dict.zh.th7 -path trainings/convenc/model_best.th7 -beam 10 -nbest 2
# you actually have to implement the solution
# <unk> 这种 道德 又 能 实现 这些 目标 。
@article{gehring2017convs2s,
author = {Gehring, Jonas, and Auli, Michael and Grangier, David and Yarats, Denis and Dauphin, Yann N},
title = "{Convolutional Sequence to Sequence Learning}",
journal = {ArXiv e-prints},
archivePrefix = "arXiv",
eprinttype = {arxiv},
eprint = {1705.03122},
primaryClass = "cs.CL",
keywords = {Computer Science - Computation and Language},
year = 2017,
month = May,
}
@article{gehring2016convenc,
author = {Gehring, Jonas, and Auli, Michael and Grangier, David and Dauphin, Yann N},
title = "{A Convolutional Encoder Model for Neural Machine Translation}",
journal = {ArXiv e-prints},
archivePrefix = "arXiv",
eprinttype = {arxiv},
eprint = {1611.02344},
primaryClass = "cs.CL",
keywords = {Computer Science - Computation and Language},
year = 2016,
month = Nov,
}
fairseq is licensed from its original repo.
Pretrained models in this repo are BSD-licensed.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent