Suggestion: Add basic problems like sentence segmentation and tokenization about nlp-progress

[stefan-it]

I implemented a sentence boundary detection system using several neural network architectures: deep-eos. (Sadly my paper was rejected at a conference so I decided to open source my implementation).

Corpora are a bit hard to find - some paper I read used Europarl. The problem is that Europarl is not 100% sentence segmented.

Another resources could be the universal dependencies datasets, because a good sentence boundary detection system should also work for non-english languages.

I'm currently working on sentence boundary detection using (universal) language models :)

[sebastianruder]

Thanks so much!
That's a great idea! I think it'd be great to make people aware that these tasks are actually not solved.

Do you know what datasets are used for sentence segmentation and tokenization by any chance? I guess we could look at what spacy uses for evaluation.

[sebastianruder]

Cool. Feel free to add them to the wish list for now then. I'll look up results once I have some time or maybe someone gets around to it first.

[pwichmann]

It'd be great to see how well basic tasks are actually solved and what the conditions are under which they still perform very poorly. I'd hope that some people then address these special cases with better techniques.

I was surprised by this simple sentence that could not get sentence-tokenized correctly. Spacy.io made 2 sentences out of it.

[pwichmann]

Same with inconsistent splitting when I use "...". This was surprising.