Chinese segmentation not correct about charabia

[sivdead]

I can make a pr to solve this if you do think this should be fixed.

[ManyTheFish]

Hello @sivdead,
you're right, using cut_for_search would increase the recall of Meilisearch by splitting words in different ways.
However, Meilisearch relies on words position for queries, and Jieba.cut_for_search doesn't give any clues on the position of each token, moreover, charabia does not support shifting tokens.
In order to support this kind of position shifting behavior, the charabia output should be changed in a tree shape for instance 永永远远是龙的传人 would be shaped as:

永永远远 ──┬─► 是 ─┬─► 龙的传人
永远 ─────┤       └─► 传人
远远 ─────┘

Which is not possible without doing a huge job,
But I have to admit that it would enhance significantly the search recall.

Thank you for your report and sorry for the time to answer,

see you!