Approaching browse/download problem about sciencefair

[blahah]

Hey, thanks for your interest! We actually just hammered out some details of this last night.

We're pre-generating a full index of open-access papers, starting with EuropePMC (should be done by tomorrow). This will be distributed with the app using dat.

The initial indexes will be generated by parsing full data dumps. For example, EuropePMC data is here: http://europepmc.org/FtpSite, and PubMed data is here: http://www.ncbi.nlm.nih.gov/pmc/tools/ftp/.

Later we will merge in other data sources (e.g. CrossRef, DOAJ, Academic Graph, etc.) and allow users to add their own.

More generally we've tackled these issues on the ContentMine project by using a combination of API access tools (see getpapers) and collections of declarative scrapers for providers that don't deposit their data anywhere useful (see quickscrape, scraperjson and journal-scrapers).

Ways to help right now include:

• watch the repo so you'll get notified when new issues are added
• take a look at the linked projects above which are more ready for contributors right now

And here's a small cactus 🌵 as a prize for creating the first issue.

[max-mapper]

@MMMalik also we were thinking of other data preprocessing we could to to make the paper metadata browsing/search more fun/valuable. One idea was to try to extract figures from the papers (using some pdf to png pipeline that was could intelligently create screenshots of the figures in the pdf, etc). If you have any ideas like that it could be a good way to start experimenting.