sentence-splitter about mosesdecoder

[hieuhoang]

better off asking the mailing list. Hardly anyone pays attention to this forum
http://mailman.mit.edu/mailman/listinfo/moses-support
please subscribe before you post

[hieuhoang]

closing this. Looks like no-one's responding to this forum

[matanox]

Why close, if it is essentially open?

[hieuhoang]

I'll reopen it, but don't be surprised if u get no response.

[matanox]

don't worry, I'll refer to it on the mailing list that you suggested :-)

[tomekd]

I know nothing about moses' sentence splitter but give a try for eserix. I used it from time to time.

[matanox]

Thanks @tomekd, do you know whether it is accommodates different languages, v.s. being just useful for English? we're looking for something covering a wide range of languages, not that the Moses script was necessarily perfect at that.

[tomekd]

Hi,

it supports the most popular languages:

• English
• French
• Spanish
• German
• Polish
• Russian
• Arabic
• Chinese
• Croatian

Notice that it's really simple tool using SRX files.

[matanox]

Well, I guess, good to learn of SRX (Segmentation Rules eXchange) now :-) Other than reading the dry spec of it, may I assume that the implied algorithm comprises a two-step flow, where first a break is matched by all the break=yes rules, and then the break may be avoided if it matches any of the break=no rules? any notable libraries that execute the rules or notable rule depos? I see version 2.0 of the standard is supposed to be "safer" and Java is lagging in regex support required for it.

Essentially the perl script here has a similar flow, although it seems to struggle with introducing extra spaces that it later needs to discard, and arguably a bit of a hack when it comes to adaptation to special domains or language registers.

[hieuhoang]

looks like the mailing list got you good responses. Closing now