intelcomph2020 / semanticscholarimporter

Project to import and save to Postgres / Parquet data from SemanticScholar.

SemanticScholarImporter

This repository provides code to save, import and process SemanticScholar dataset.

Semantic Scholar is a search engine for research articles powered by the Allen Institute for Artificial Intelligence.

## 1. Download raw data

• Copy and rename config_default.cf into config.cf adjusting options as needed. You just need to fill in the first section [downloadS2]:

  • dir_data is the path to the directory where you want to save the raw data
  • S2_API_Key are your credentials for downloading the Semantic Scholar dataset. You should get in contact with the Allen Institute to get them
  • version is the version you want to download (default value latest is normally what you will want)

• Run the following command line

python downloadSS.py

2. pySpark/SemanticScholar_to_parquet.ipynb

This notebook contains all the code to transform original json files to parquet files. This notebook should be adapted to run in your pySpark cluster

It creates the following tables in parquet format:

• papers: information related to papers
• authors: maps author id to their name
• paper-author: maps each paper to its authors
• citations: source paper to referenced paper

Additionally, the notebook includes a section to create some domain specific datasets for Artificial Intelligence and Cancer

semanticscholarimporter's Issues

Is it really necessary to have a different class for postgres than for MySQL?

The problem of having to classes, one for mysql and another for postgres is that changes done in one of them should be migrated to the other. One ends up having two different classes with many functions in common.

Would it be feasible to have a parent class with common functions, and then inherit with two different classes ?

Priority of this issue is low ... we need to discuss when and if this is implemented at all

Add option to create indexes, and don't do it while creating schema

If indexes are created from the beginning, data loading on the postgres db will be very slow. It is better to load data first, and then create indexes once the data is ready. Two choices are given:

• Preferred: Create indexes on each table after ingesting all data relevant to that table
• Other: Create a new option in the menu to create indexes at the very end

Enrich Author metadata with other information

We can enrich author information with:

• influentialCitationCount: Very soundy, but probably costly to obtain, since they need to be crawled from SS using their API.
• ORCID data
• Affiliation and/or whether the author has Spanish affiliation or not

The priority of this enhacement is low

Create option to enrich citation table

Use Semantic Scholar API to enrich citation table with the following information:

• isInfluential
• BackIntent
• MethodIntent
• ResultIntent

This information is very useful, but probably costly to obtain. So:

1. Check if part of this information could be obtained online, or if we have access to the software they use and it is feasible to run it
2. Implement the possibility of enriching using the API (is this already implemented?). Add for this second case the possibility of filtering with some paper metadata. E.g., enrich only information from papers beloinging to a certain S2 Field.