jeongukjae / nori-clone

Standalone Nori (Korean Morphological Analyzer)

nori-clone

Standalone Nori (Korean Morphological Analyzer in Apache Lucene) written in C++.

blog post (written in Korean)

Introduction

ElasticSearch provides high-quality/performance Korean morphological analyzer nori. But nori's code is strongly coupled with the Lucene codebase, and nori is written in Java that is the main language in the Lucene project. So, it's hard to use nori standalone in Python or Golang with the same performance. Therefore, I re-implemented almost the same algorithms with nori in Lucene using C++ for the portability and usability.

Usage

This project is written in C++, but also provides Python and Golang binding.

• C++ docs
• Python docs
• Golang docs

Pre-built dictionaries

A dictionary/ directory is for the pre-built dictionary files that is used for distribtion and test cases. For now, there are two pre-built dictionaries, lagacy and latest.

• legacy dictionary does not normalize inputs, and built with mecab-ko-dic-2.0.3-20170922 that is same with original nori.
• latest dictionary normalizes the inputs with the form NFKC, and built with mecab-ko-dic-2.1.1-20180720.

Performance

For more details, check out tools/benchmark.

Differences with original nori

Check out tools/comparison.

For the contributors

Check out CONTRIBUTING.md

References

• https://www.elastic.co/kr/blog/nori-the-official-elasticsearch-plugin-for-korean-language-analysis
• https://github.com/apache/lucene/tree/main/lucene/analysis/nori
• https://gritmind.blog/2020/07/22/nori_deep_dive/
• https://github.com/gritmind/python-nori

nori-clone's Issues

add punctuation handling feature

add benchmark information

• vs nori in lucene
• vs py-nori
• vs MeCab
• vs kiwi

Unifying dictionary format to single file

In the current implementation, we have to bring several files to use nori-clone, and most of the file is in the protobuf format. It would be better to bring just a single file.

add c_api packaging action

for the go binding

platform

• macosx amd64
• linux amd64

add script to validate tokenizer's output

with nori in lucene

add windows support

add PyPI deployment action

add golang binding

check outputs with original nori

update python interface

The current python interface is not convenient to use.

import nori

dictionary = nori.Dictionary()
dictionary.load_prebuilt_dictionary("./dictionary/latest-dictionary.nori")
dictionary.load_user_dictionary("./dictionary/latest-userdict.txt")
tokenizer = nori.NoriTokenizer(dictionary)

result = tokenizer.tokenize("이 프로젝트는 nori를 재작성하는 프로젝트입니다.")

for token in result.tokens:
    print(token.surface)

I think the API should be changed as follows.

import nori

tokenizer = nori.NoriTokenizer()
tokenizer.load_prebuilt_dictionary("./dictionary/latest-dictionary.nori")
tokenizer.load_user_dictionary("./dictionary/latest-userdict.txt")

result = tokenizer.tokenize("이 프로젝트는 nori를 재작성하는 프로젝트입니다.")

for token in result.tokens:
    print(token.surface)

add python guide

Cannot load NFKC Normalizer in Linux env

Cannot get NFKC Instance. icuErrorName: U_FILE_ACCESS_ERROR

unknown char bugs

input: 도로ㆍ지반ㆍ수자원ㆍ건설환경ㆍ건축ㆍ화재설비연구
expected:

 도로, MORPHEME, NNG, NNG
 ᆞ, MORPHEME, UNKNOWN, UNKNOWN
 지반, MORPHEME, NNG, NNG
 ᆞ, MORPHEME, UNKNOWN, UNKNOWN
 수자원, COMPOUND, NNG, NNG
 ᆞ, MORPHEME, UNKNOWN, UNKNOWN
 건설, MORPHEME, NNG, NNG
 환경, MORPHEME, NNG, NNG
 ᆞ, MORPHEME, UNKNOWN, UNKNOWN
 건축, MORPHEME, NNG, NNG
 ᆞ, MORPHEME, UNKNOWN, UNKNOWN
 화재, MORPHEME, NNG, NNG
 설비, MORPHEME, NNG, NNG
 연구, MORPHEME, NNG, NNG

current:

 도로, MORPHEME, NNG, NNG
 ᆞ지반ᆞ수자원ᆞ건설환경ᆞ건축ᆞ화재설비연구, MORPHEME, UNKNOWN, UNKNOWN

add Doxygen page

More proper POS types for terms in the user dictionary

It would be better if we can assign more proper POS types for terms in the user dictionary. Currently, all user dictionary's terms are treated as NNG.

We can find the required right-ids and left-ids for the user dictionary at right-id.def and left-id.def files.

3533 NNG,*,*,*,*,*,*,*
3534 NNG,*,F,*,*,*,*,*
3535 NNG,*,T,*,*,*,*,*
3536 NNG,지명,F,*,*,*,*,*
3537 NNG,지명,T,*,*,*,*,*
3538 NNP,*,F,*,*,*,*,*
3539 NNP,*,T,*,*,*,*,*
3541 NNP,인명,*,*,*,*,*,*
3542 NNP,인명,F,*,*,*,*,*
3543 NNP,인명,T,*,*,*,*,*
3544 NNP,지명,*,*,*,*,*,*
3545 NNP,지명,F,*,*,*,*,*
3546 NNP,지명,T,*,*,*,*,*

add synonym filter

retrieve default right id information from `right-id.def`

add reading form feature

add semantic group info in output

example)

term: 서울,1782,3538,7662,NNG,장소,T,서울,*,*,*,*
semantic group: 장소