open-numbers / ddf--unpop--world_population_prospects

UN World Population Prospects

This dataset contains all indicators from UN World Population Prospects.

Check the concepts file for a list of all indicators.

Data sources summary

We use CSV files from the download page. Locations comes from the Location Metadata File.

Notes

• there are different methods for projections for datapoints for future years. We only keep the Medium projection.
• We compute total dependency ratio and sex ratio by age groups in ratios folder.

How to run the script

You should download all csv files listed in etl/scripts/urls.txt from WPP and put them into the etl/source directory. Then you can run the etl.py to create the dataset.

ddf--unpop--world_population_prospects's Issues

Split up datapoint files per country

Right now the datapoint files are > 100MB which is above the Github limits.

oh yes, two datapoint files, 127MB and 293MB. Github don't allow to send such big files (edited)
we can try git-lfs https://git-lfs.github.com/
but this is a paid service https://help.github.com/articles/billing-plans-for-git-large-file-storage/ for usage > 1GB

https://help.github.com/articles/what-is-my-disk-quota/
https://help.github.com/articles/working-with-large-files/

Let's split datapoints up in smaller files per geo. We need this anyway for DDFcsv reader not to read a massive file of all geo's for showing only one geo.

Files names should follow the pattern
ddf--datapoints--<indicator>--by--geo-<geo>--<other dimensions>.csv

e.g.
ddf--datapoints--population--by--geo-usa--gender--age--year.csv
ddf--datapoints--population--by--geo-swe--gender--age--year.csv
ddf--datapoints--population--by--geo-chn--gender--age--year.csv

Concepts name incorrectly contain gender

97608e5#diff-29c950e46477941fb6e5504b45b48b21

In the latest autogenerated commit, these concepts names say "Female" while the concept is used for any gender (male, female and both sexes). Is this because WPP doesn't contain a gender-agnostic concept name you can use, so you just pick one from the male/female/both sexes variable names?