juzraai / cordis-projects-crawler

Retrieves data of all projects on CORDIS

Crawler for CORDIS projects

Version: 2.0.0 · Author: Zsolt Jurányi (web, github)

---

What's this?

This is a small and easy-to-use command line utility which can help you grab CORDIS projects' information automatically, along with related publications from OpenAIRE. You tell the program which projects you are interested in, and the software downloads the files which contain the required data. It can parse them and generate TSV or MySQL export.

How to use it?

See the User guide.

What's new in version 2?

The old v1.3.0 is not working anymore on the updated CORDIS website.

• I rebuilt the crawler from scratch, switched from Java to Kotlin.
• Extracts data from CORDIS XMLs and OpenAIRE API instead of messy HTML pages and encoded JSON strings.
• Can parse information of all projects, from oldest to newest.
• There are new ways to tell the crawler which projects are needed.
• Modular architecture helps developers to extend/alter functionality of the crawler.
• Sexy documentation using Docsify

Future plans, ideas

• Configuration: chunk (batch) size (-b)
• Configuration: request sleep (-z)
• Seed: RCN list file (-s list:<filename>)
• Export: RCN list file (-l)
• Seed: CORDIS bulk file (CSV, XLSX, ZIP) (-s bulk:<filename>)
• Reader: CORDIS bulk file (CSV, XLSX, ZIP), after default file cache
• Crawl project documents (webItem) (-d)
• Crawl result XMLs (-r)
• Generate "unified view" export (-u)
• Module registry: addBefore(Class) method
• Existing TSV exporters: reduce code redundancy

Interested as a developer?

Read the Developer guide and contributing instructions.

This project is licensed under MIT License.