nerevu / hdxscraper-who

WHO Global Health Observatory Data Collector

WHO Global Health Observatory Data Collector

HDX collector for WHO Global Health Observatory Data.

Introduction

hdxscraper-who operates in the following way:

• Downloads Global Health Observatory Data json data
• Groups the rows by indicator
• Dynamically creates a separate database table for each group

With hdxscraper-who, you can

• Save WHO Data to an external database
• Create CKAN datasets with externally generated CSV files
• Update resources previously uploaded to CKAN with new metadata

View the live data

Requirements

hdxscraper-who has been tested on the following configuration:

• MacOS X 10.9.5
• Python 2.7.10

hdxscraper-who requires the following in order to run properly:

• Python >= 2.7 (MacOS X comes with python pre-installed)

Setup

local

(You are using a virtualenv, right?)

git clone https://github.com/reubano/hdxscraper-who.git
pip install -r requirements.txt
manage setup

ScraperWiki Box

rm -rf tool
git clone https://github.com/reubano/hdxscraper-who.git tool
cd tool
make setup

Usage

local

manage run

ScraperWiki Box

source venv/bin/activate
screen manage -m Scraper run

Now press Ctrl-a d. The results will be stored in the file scraperwiki.sqlite.

view all available commands

manage

Upload tables to HDX/CKAN

upload to production site

manage upload

upload to staging site

manage upload -s

Update tables on HDX/CKAN with new metadata

update dataset on production site

manage update

update dataset on staging site

manage update -s

Update ScraperWiki box with new code

cd tool
make update
source venv/bin/activate
screen manage -m Scraper run
# Now press `Ctrl-a d`

Configuration

hdxscraper-who will use the following Environment Variables if set:

Environment Variable	Description
CKAN_API_KEY	Your CKAN API Key
CKAN_PROD_URL	Your CKAN instance remote production url
CKAN_REMOTE_URL	Your CKAN instance remote staging url
CKAN_USER_AGENT	Your user agent

Creating a new collector

If you would like to create collector or scraper from scratch, check out cookiecutter-collector.

pip install cookiecutter
cookiecutter https://github.com/reubano/cookiecutter-collector.git

Contributing

Code

1. fork
2. commit
3. submit PR
4. ???
5. PROFIT!!!

Document

• improve this readme
• add comments to confusing parts of the code
• write a "Getting Started" guide
• write additional deployment instructions (Heroku, AWS, Digital Ocean, GAE)

QA

1. follow this guide and see if everything works as expected
2. if something doesn't work, please submit an issue

License

hdxscraper-who is distributed under the MIT License.