Words do not get divided properly when small letters (捨て仮名) are included in word about mecab

[polm]

Hey, this is not a bug but just an example of the limitations of Mecab. Depending on the settings for unknown words, some sequences of words can be combined together as an UNK (未知語) instead of being split. It's particularly easy to cause this if the sequences of words you're using aren't similar to the kind used in the training data for the dictionary (mostly newspaper-article type stuff). This is because Mecab uses not only the existence of words in the dictionary, but also transitions between different parts of speech when calculating the best place to split words.

In particular, Mecab just treats sutegana the same as kana. You can see this in the character class definitions in the char.def file distributed with ipadic, which is the dictionary you appear to be using.

You can read more about unk processing here.

[rolzy]

I have found another example, this time where the 捨て仮名 is not at the front but still included:

ありがとうございますなんかはそうですね
ありがとう      感動詞,*,*,*,*,*,ありがとう,アリガトウ,アリガトー
ござい  助動詞,*,*,*,五段・ラ行特殊,連用形,ござる,ゴザイ,ゴザイ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
なんか  助詞,副助詞,*,*,*,*,なんか,ナンカ,ナンカ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
そうですね      フィラー,*,*,*,*,*,そうですね,ソウデスネ,ソーデスネ
EOS
ありがとうございますなんかはそうですねちょっと
ありがとう      感動詞,*,*,*,*,*,ありがとう,アリガトウ,アリガトー
ござい  助動詞,*,*,*,五段・ラ行特殊,連用形,ござる,ゴザイ,ゴザイ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
なんか  助詞,副助詞,*,*,*,*,なんか,ナンカ,ナンカ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
そうですね      フィラー,*,*,*,*,*,そうですね,ソウデスネ,ソーデスネ
ちょっと        副詞,助詞類接続,*,*,*,*,ちょっと,チョット,チョット
EOS
ありがとうございますなんかはそうですねちょっとやっぱ
ありがとう      感動詞,*,*,*,*,*,ありがとう,アリガトウ,アリガトー
ご      接頭詞,名詞接続,*,*,*,*,ご,ゴ,ゴ
ざいますなんかはそうですねちょっとやっぱ        名詞,一般,*,*,*,*,*
EOS

[rolzy]

Thank you very much for your reply!