semio / ddf--gapminder--gapminder_world Goto Github PK

Failed to lookup placeholders in Windows

creating datapoints...
concept not found: ———————————————————————
concept not found: ———————————————————————
concept not found: ———————————————————————
concept not found: ———————————————————————
concept not found: ———————————————————————
concept not found: ———————————————————————
concept not found: ———————————————————————
concept not found: ———————————————————————
concept not found: ———————————————————————
concept not found: ———————————————————————
concept not found: ———————————————————————
concept not found: ———————————————————————
concept not found: ———————————————————————
concept not found: ———————————————————————

It seems East Timor has been renamed over the years from East Timor to Timor-Leste in the http://www.emdat.be dataset.

That's why in Gapminder World google spreadsheets it is featured with both those names on separate rows in emdat.be data
https://docs.google.com/spreadsheets/u/1/d/1EMSP8rthB6yAxj3GtPAcssfP0HHPfujRS0YDPmD1NRY/pub
https://docs.google.com/spreadsheets/u/1/d/1_UEhuCQeH5MySwuOKmawjRNeQkwP2vJx0rZb7Wgq2wE/pub#

more should be added as we find them

If you look closely, you can see that there is no overlap of numbers > 0.
Somewhere between 2003 and 2007 the emdat.be must've changed names.
Fill out Timor-Leste here and you see that combined numbers are correct: http://www.emdat.be/country_profile/index.html

Both names are translated to tls when turning Gapminder World to DDFcsv. Therefore, there is tls datapoints for both East Timor and Timor-Leste and thus duplicate keys.

How to solve:

1. Update source, combining East Timor and Timor-Leste data to one row
2. Make script smart so it merges the two
3. Keep script dumb and make an exact copy of data as it is: Make sure there are two separate entity ids for East Timor and Timor-Leste. Though this keeps the error in the dataset (not sound).

This has no priority as ddf--cred--em_dat should overwrite this data correctly in SG.
This would be purely to make this historic dataset valid and sound.

Traceback (most recent call last):
File "main.py", line 159, in
r = main(source_dir, ddf_dir, vizabi_dir, make)
File "main.py", line 117, in main
md = generate_metadata(cs, concepts_, mdata, area, ddf_dir)
File "/Users/semio/src/work/Gapminder/ddf--gapminder--gapminder_world/etl/script/vizabi.py", line 121, in generate_metadata
indb['indicatorsDB'][key]['color'] = eval(indb['indicatorsDB'][key]['color'])
File "", line 1, in
NameError: name 'false' is not defined

continuous concepts from dont-panic-poverty data set didn't in ddf--concetps yet.

1. Micronesia is a geographic region (like southern Europe)
2. Federated States of Micronesia is a country in Micronesia, among other countries (like Italy is in Southern Europe)
3. Gapminder World google spreadsheets version uses full country names
4. Gapminder World ddfcsv version uses entity ids
5. To translate country names to entity ids we use this file https://github.com/Gapminder/waffle-server-importers-exporters/blob/world-legacy-with-data/data/synonym/country_synonyms.xlsx
6. The file contains Micronesia as a synonym for Federated States of Micronesia and does not contain an entry for Micronesia itself.
7. Indicators of UNPOP have data for both Micronesia and Federated States of Micronesia
8. Translating these indicators' names to entity id's gives two datapoints for Federated States of Micronesia, because of the automatic translation translating Micronesia stats to Federated States of Micronesia.

How to solve:

1. Remove Micronesia as a synonym for Federated States of Micronesia and add an entity id for Micronesia Region (and other regions in dataset which are not in geo-entities now).
2. Change name of Micronesia to something less ambiguous (Micronesia (Region)) in source

I'd go for 1.