kmario23 / kenlm-training Goto Github PK

Hi, I got error like in title when running this command:
./kenlm/bin/build_binary trie bible.arpa bible.binary

everything was good until this step, no sign of out of memory or something like that, it crashed immediately
any suggestion?

p/s: if I cant solve this problem, can I just use the arpa file (without trie)? it give the same score with the demo sentence in the README

I am trying to make an arabic language model the problem is the characters are not understandable by many frameworks so I use a simple conversion to map each character to English characters but arabic characters are more than English so there are capital letters in the corpus that will change the meaning if lowered. Will be that an issue to use capital letters?

Hi, a great thanks to the post on how to install kenlm. i have build bible.arpa and able to get the scores of the sentence using model.score(), now i would like to get the suitable n number of words when a character/sentence is given, how to get that.
one more is what are all the functions available for this model, where can i get those functions or arguments

After i run the command :bzcat bible.en.txt.bz2 | python preprocess.py | ./kenlm/bin/lmplz -o 3 > bible.arpa,

ERROR:bash: ./kenlm/bin/lmplz: No such file or directory
Traceback (most recent call last):
File "preprocess.py", line 6, in
print(' '.join(nltk.word_tokenize(sentence)).lower())
BrokenPipeError: [Errno 32] Broken pipe.

Hello
Please I am following this tutorial to create my French Language model : https://github.com/kmario23/KenLM-training
But when I type this cmd :
bzcat ./data_final/vocabulary.txt.bz2 | python preprocess.py | /home/innovation/kenlm/bin/lmplz -o 3 > myvocabulary.arpa

I get the following error :

print(' '.join(nltk.word_tokenize(sentence)).lower())
BrokenPipeError: [Errno 32] Broken pipe
Erreur de segmentation (core dumped)

Hi,
I have successful run all those steps in README and have bible.arpa bible.binary but there is no trie file
How can I generate trie? I cant find any tutorial about this

The original repo is: https://github.com/kpu/kenlm (1600 stars) but the instructions are to clone https://github.com/vchahun/kenlm.git which is a fork with only 5 stars and is 1000 commits behind master.