C++ implementation of Neural Machine Translation of Rare Words with Subword Units.
Compile with:
g++ -std=c++11 -pthread -O3 fast.cc -o fast
./fast
usage: fastbpe <command> <args>
The commands supported by fastBPE are:
getvocab input1 [input2] extract the vocabulary from one or two text files
learnbpe nCodes input1 [input2] learn BPE codes from one or two text files
applybpe output input codes [vocab] apply BPE codes to a text file
fastBPE also supports stdin inputs for getvocab
and learnbpe
. For instance, these two commands are equivalent:
./fast getvocab text > vocab
cat text | ./fast getvocab - > vocab
But the first one will memory map the input file to read it efficiently, which can be more than twice faster than stdin on very large files.
./fast learnbpe 40000 train.de train.en > codes
./fast applybpe train.de.40000 train.de codes
./fast applybpe train.en.40000 train.en codes
./fast getvocab train.de.40000 > vocab.de.40000
./fast getvocab train.en.40000 > vocab.en.40000
./fast applybpe valid.de.40000 valid.de codes vocab.de.40000
./fast applybpe valid.en.40000 valid.en codes vocab.en.40000
./fast applybpe test.de.40000 test.de codes vocab.de.40000
./fast applybpe test.en.40000 test.en codes vocab.en.40000