Custom words dictionary about deepcut

[titipata]

Hi @venzen I can confirm that deepcut cannot do the text correction or spelling correction. You may have to create a function (after OCR) to deal with these corrections before performing the tokenization.

For the second question, I think deepcut should generalized enough to parse new unseen words. However, you can put list of words in the dictionary to cover some cases that you think that it may tokenize wrongly.

Hope this help a bit! Maybe someone can follow this issue too!

[titipata]

There was a paper for spelling correction by Ekapol and team which may help you quite a bit: https://ieeexplore.ieee.org/document/9145483. I'm not sure if they provide open-source implementation somewhere.

[venzen]

@titipata Thank you for your response. I started implementing word similarity checking before I saw your reply. Can use difftool.SequenceMatcher() with a text file of Thai words and find correct spellings.

As you recommend, I then add new words to this file and add it as a deepcut custom dictionary. For example deepcut was tokenizing 'ความนํา' as ['ความ', 'นํา'] and after adding the word to the custom dictionary it is correctly tokenized as ['ความนํา'].

Thank you, also, for sending the link to the paper about Thai spelling correction. I will read and give feedback if I find a solution.

[venzen]

Unexpected behavior from deepcut: I am passing a custom dictionary that contains both the words 'หรือ' and 'อิริยาบถ'. Each word is a separate entry on its own line and without whitespace. There is a newline ('\n') after each entry. So we would expect deepcut will tokenize each word, correct?

The string is: หรืออิริยาบถน้อย มีคู้เข้า เหยียดออก....

deepcut fails to segment and returns a single list item for 'หรืออิริยาบถ':

['หรืออิริยาบถ', 'น้อย', ' ', 'มี', ... ]

Any idea what is happening in this case?

EDIT: I should add that my custom dictionary includes 19,000 Thai words. Some of them are compound words. Perhaps this is causing strange behavior.

[venzen]

The issue was that the custom dictionary contained duplicate words (words also present in the deepcut dictionary). When I made a new blank custom dictionary deepcut works as expected.