Can it analyse Chinese? about mallet

[mimno]

Mallet is not language-specific, but does not support Chinese tokenization. You would need to run something like the Stanford Chinese segmenter on input text before importing.

[chandrasg]

Hi @mimno , I applied text segmentation on chinese text which I am using as input to mallet.

My input looks like this:
id1 id2 【 食尚 玩家 冰糖 红烧肉 】 ） 五花肉 切块 ， 葱

I am trying to capture both english and chinese tokens. Is this the right way to do it?
../mallet-2.0.8/bin/mallet import-file --input inputfile.txt --token-regex '([\p{L}\p{M}]+|[\p{IsHan}]+)' --output inputfile.mallet --keep-sequence --remove-stopwords --stoplist-file stopwords-zh.json

Thanks!