What about other languages? about nlp-progress

[NirantK]

Hey @sebastianruder @Hrant-Khachatrian ,

We maintain datasets and tools for several languages e.g. Korean, Arabic at awesome-nlp

To avoid duplication of effort, we can either

• Add the State of the Art results there (we have a research section, which we'll remove shortly and instead point here)
• Or we can move the entire non-English content to this repository and reorganize as you would like
• Other options?

Let's avoid duplication either way, what do you think?

[sebastianruder]

Hey, I totally agree. The most important thing for me is not to duplicate effort. I've been following awesome-nlp and really like the collection of tools, particularly across languages.
For me personally, knowing about a dataset without links to papers or results on that dataset, however, hasn't been that useful.
For that reason, what would make the most sense for me would be to add non-English datasets and results to this repo. I'd love to add you as collaborators/maintainers to this repo if that doesn't seem like too much additional work.

[sebastianruder]

Yes! Including other languages is definitely on the road map!

I'm not sure what's the best way to include them at the moment. We could break out the results first per task and then per language. Breaking everything out per language first might make it hard to find things. Maybe with some better visualization (which we're working on at the moment), including other languages would be easier. Do you have any suggestions?

[NirantK]

Sure, happy to contribute.

We don't maintain results/papers for the libraries and datasets at awesome-nlp yet.

I will start by adding different language datasets, and we can add results as and when we find them. Does that sound good?

How do we handle libraries or tools?

In parallel, I'll add a link to nlpprogress.com to every language that is migrated here.

[PiotrCzapla]

Guys, we have results and data sets for polish nlp, where should we post them, this is for NER and for Language Modeling. How about we add a headline to each English pages as the most advanced language with links to other languages?
I will propose a change in a second.

[sebastianruder]

Let's have the discussion what is the best format for adding other languages here. @PiotrCzapla summarized this well in #105:
Either we have a file for each task linking to the task in each language:

• language_modeling.md
  • ar_language_modeling.md
  • pl_language_modeling.md

Or we'd have a file for each language linking to each task in the language:

• ar.md
  • language modeling
  • sentiment analysis

I think this mainly depends on what is the preferred way that people will look up tasks / results. Personally, I think the preferred setting is to look for one's own language and then at the tasks in that language. As long as we don't have too many tasks and languages yet, we could thus simply extend the main README.md file with additional languages and the corresponding tasks, e.g.

• Arabic
  • Language modeling
  • Sentiment analysis
  • etc.
• English
  • Language modeling
  • Sentiment analysis
  • etc.

What do you think?

[NirantK]

This sounds good to me!

Since most libraries are around POS tagging, tokenizers etc. in several languages, will start there over coming weekend.

[PiotrCzapla]

Sounds good as long as we can keep English on the top of the file to keep the most up to date research on neural networks easily accessible. But I’m not sure if we won’t offend someone that way.

…

On 18 Sep 2018, at 5:22 AM, Nirant ***@***.***> wrote: This sounds good to me! Since most libraries are around POS tagging, tokenizers etc. in several languages, will start there over coming weekend. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[sebastianruder]

Yep, good point. Let's keep English on the top.
@NirantK, given your great work on awesome-nlp and other NLP projects, would you like me to add you as a collaborator to the repo?

[NirantK]

Sure, happy to help.

Even then, I'll start by raising a few PR's so that we can workout any accidentally left out details.