surafelml / afro-nmt

LOW-RESOURCE NEURAL MACHINE TRANSLATION: A BENCHMARK FOR FIVE AFRICAN LANGUAGES

Surafel Melaku Lakew

Scientist at Amazon AI

• Neural Machine Translation
• Self-Supervised Learning from Unlabeled Data
• Natural Language Processing, Deep Learning, Machine Learning

afro-nmt's Issues

Error while APPLYING SP MODEL ON [train]

i'm working on sw-en translation. After running build-training-data.sh the train.src looks like this : <2eng> Je, utatupa hii?
While running preprocess.sh getting the below error i.e. not able to read '<' characters as per my understanding.

Traceback (most recent call last):
File "/gpfs-volume/Afro-NMT/scripts/spm-subword.py", line 95, in
main()
File "/gpfs-volume/Afro-NMT/scripts/spm-subword.py", line 79, in main
encode(args.spm_dir, args.in_file, args.src, args.op_file)
File "/gpfs-volume/Afro-NMT/scripts/spm-subword.py", line 36, in encode
print(' '.join(spm_spp.EncodeAsPieces(line.strip())), file=outfile)
UnicodeEncodeError: 'ascii' codec can't encode character '\u2581' in position 0: ordinal not in range(128)

Can you help how it can be resolved.