NLP Progress Graph about nlp-progress

[sebastianruder]

Hey,
I think that's a great idea! If we could include an up-to-date picture of the overall progress (in terms of new results) across all tasks, that would be awesome!
For the graph, I guess the main thing we want to track is new results that are added. As a PR can include multiple commits, it might get confusing if we ask people to include "new result" in a commit. So maybe asking to include "new result" in the title of the PR might make more sense?

[sebastianruder]

Yep, good idea. I've created one here. I'll link to this more prominently.

[nirmalsinghania2008]

Yeah. that's right.
Also, markdown doesn't provide dynamic charts otherwise we could've had something like "time period" over which the results are added. :(

[nirmalsinghania2008]

Maybe we can move to gh-pages.
https://nirmalsinghania2008.github.io/NLPprogress/
Looks good in my opinion.

[sebastianruder]

So, do you think if we add the instruction for people to include "new result" in the title of PR if the PR contains a new result, you could create a chart based on the PRs?

[nirmalsinghania2008]

Yep. I think so

[sebastianruder]

👍 I've just added a note for people to include "new result" in the title (bd91d2a) and will point future PRs to that.

[stared]

Plots may be very important, vide Performance section in Keras vs PyTorch vs plain tables.

And see also Measuring the Progress of AI Research by EFF, which you may already know.

Recreating with each addition is time consuming. At the same, if instead of MD files, data were stored in some semantic files (e.g. for human readability YAML), it would be easy to create HTML plots (as in: I am happy to create such). Jekyll has support for YAML, which I already (ab)use (for list of projects, conferences, etc).

With that we can create both a table (maybe even sortable) and plots. Plus, separating semantic (pun kind of intended) data from its presentation is a plus.

[sebastianruder]

Hi Piotr, thanks a lot for contributing to the discussion here. I had a look at EFF initially, but opted to keep it simple for the beginning.
I agree that having plots / graphs is a good idea. Given that we're not currently breaking out things by exact date of publication at the moment, my main concern is how much value they'll add vs. increased complexity, particularly for the rarer tasks. That doesn't mean that I don't want plots; I just want to make sure we get them right. :)
If you have time, would you mind taking one task and creating a plot for it, which we can use as a reference for further discussion?

[stared]

Sebastian, sure. I would be happy to give it a try this week.

I understand that EFF-style would be harder to maintain and collaborate on. For YAML, it won't add much complexity or confusion. Instead of tables, there will be something like:

- paper: "Blablisation of tokens"
  year: 2018
  score: 78.9
  link: https://dx.doi.org/1234.1234.1234
  implementation: https://github.com/aaa/bbb

However, how to locally build the site? (While I am familiar with Jekyll, the file structure is somewhat strangely flat). jekyll serve does not generate any index.html file. Is it some GitHub-only style?

[sebastianruder]

You mean the site at http://nlpprogress.com/?
That's just GitHub pages. In your fork, you can just go to Settings/GitHub Pages and then just switch it on.

[stared]

I see (i works via GitHub Pages after deleting CNAME, but due to some installation hell cannot set it up locally). I am not sure this kind of Jekyll project supports data (cf. more generic structure of a Jekyll project, as in https://github.com/stared/stared.github.io).

If I come up with sth, I will let you know (most likely via PR :)).

[sebastianruder]

Yeah, I'm aware of Jekyll. I wanted to keep it as simple as possible for now to make maintenance and collaboration easy.
I'm open to using Jekyll, if we can still make it easy for people to contribute. Would love to see what you come up with. :)

[stared]

@sebastianruder Added a Pull Request #64 (right now mostly for viewing; if it works, we can decide on the exact form of entries and I am happy to rewrite quite a few things).

As a side note, the project may benefit from some structure, and splitting index.md (automatically listing all pages) from README.md (not there is some UX overlap between GitHub files and the website). See: https://github.com/stared/NLP-progress/tree/feat-restructure

I didn't include these changes as they are controversial, but not necessary to introduce plots. Let me know if it is also something you would like to consider (IMHO much cleaner structure).

[sebastianruder]

Thanks! I agree that a restructuring makes sense. Automatically listing the table of contents is also a good idea.

[stared]

Great, so I will try to incorporate that as well.