po3rin / chinormfilter

Filter synonym files written in lucene format to avoid duplication with Sudachi normalization.

chinormfilter

Filter synonym written in lucene format to avoid duplication with Sudachi normalization. Mainly used when migrating to sudachi analyzer.

Usage

$ chinormfilter tests/test.txt -o out.txt

filtered result is following.

レナリドミド,レナリドマイド
リンゴ => 林檎
飲む,呑む
tlc => tlc,全肺気量
リンたんぱく質,リン蛋白質,リンタンパク質

↓ filter

レナリドミド,レナリドマイド
tlc => tlc,全肺気量

Specify system dict

$ chinormfilter tests/test.txt -s full -o out.txt

Use Custom Dict

Specify dict via sudachi.json

$ chinormfilter tests/test.txt -s sudachi.json -o out.txt

TODO

• custom dict test