everypolitician / everypolitician-data

data for national legislatures worldwide

EveryPolitician

Data about every national legislature in the world, freely available for you to use

• everypolitician.org — data | about
• Report an issue

Repo summary

These are some of the key repos in the EveryPolitician family. There are others.

• everypolitician (this repo): contains no code, but is where issues/tickets for the whole project live

• everypolitician-data: where the data is stored -- but if you want to download it, get it from:

  • human? go via the EveryPolitician website
  • program? use the RawGit CDN, via links in countries.json, which we explain here

• viewer-static: the live website http://everypolitician.org (gh-pages)

• viewer-sinatra: Sinatra app for generating a dynamic version EveryPolitician website

• webhook-manager: sends out EveryPolitician WebHooks: register your URL here!

• everypolitician-docs: documentation at http://docs.everypolitician.org/ (gh-pages)

• rebuilder rebuilds data from source

• libraries for easily manipulating EveryPolitician data (useful for all devs, but we use the Ruby ones ourselves, of course!):

  • Ruby: everypolitician-ruby and everypolitician-popolo.
  • Python: everypolitician and everypolitician-popolo

• handy gems we use when getting the data: wikidata-fetcher, wikisnakker, twitter_username_extractor, facebook_username_extractor, twitter_list, scraped_page_archive

• gender-balance: repo for the Gender Balance website that crowdsources gender data for EveryPolitician

• data_pr_change_summarizer: code used by the bot to review a data PR and leave a helpful summary as a comment

The repos for many of our scrapers are kept separately in github.com/everypolitician-scrapers.

Technical blog

The EveryPolitician bot's own page is a good jumping-off point to lots of semi-technical explanations of what's going on (it has its own blog on Medium). For example:

• how the website is built (spoiler: viewer-sinatra → viewer-static)
• how webhooks are used (you can easily register your app!)
• how the scrapers run (many live on morph.io)

The bot is on twitter as @everypolitician

Contributing

If you have data for us, or know where to get it, please read our page about how to contribute.

Team

EveryPolitician is a mySociety project.

everypolitician-data's Issues

Samoa

Official: http://www.parliament.gov.ws/new/members-of-parliament/member/

Wikipedia:

• https://en.wikipedia.org/wiki/15th_Samoan_Parliament
• https://en.wikipedia.org/wiki/14th_Samoan_Parliament

Namibia

Looks like a good candidate to do soon as there are 6 historical terms:

http://www.parliament.gov.na/index.php?option=com_contact&view=category&id=104&Itemid=1516

Associate colours with each Party/Group

So that we'll be able to draw little hemicycles etc later

(e.g. https://github.com/KohoVolit/widget-hemicycle)

Introduce a way of flagging data as not ready for publishing

Currently the publish process simply takes all final.json files and submits them to https://huboard.com/mysociety/popolo-viewer-sinatra — but these may not be ready for display yet. There should be a way of flagging each directory as publishable or not.

Clone twitter details into multiple places

These tend to appear sometimes under links and sometimes under contact_details. They should probably in both: as a full http://twitter.com/name version in links, and the @name version in contact_details. It might also be worth storing the underlying ID under identifiers too, for cases where they change their name (e.g. in the UK when they add/remove "MP")

Include Finland

Finland has been grandfathered into the live data, but doesn't exist here.

Allow multiple Sources

We will often want to link to multiple sources of data for a single country.

Ethiopia

Ethiopia has an election on May 24th, so it would be useful to scrape the existing data before then in case it disappears afterwards.

Uzbekistan

http://www.parliament.gov.uz/uz/structure/deputy/search1.php?id=%D0%90

In English:
http://www.parliament.gov.uz/en/structure/deputy/search1.php?id=+

Rwanda Former

http://www.parliament.gov.rw/chamber-of-deputies/members-profile/former-chamber-of-deputies/ — though doesn't seem to say when they were members…

Portugal

Official: http://www.parlamento.pt/DeputadoGP/Paginas/Deputados.aspx

(Includes info on historic terms for current Deputies, but I can't find information on historic deputies)

Already scraped by http://demo.cratica.org/deputados/

Belarus

Scrape the Belarus data.

Merge Botswana sources

The official source and the Wikipedia source are each different — merge them.

Germany: better IDs

We clean IDs to after the last / for other countries, but here they're delimited by :s.

South Sudan

Official: http://www.goss.org/index.php/legislative-assembly/honourable-members

Add historic name fields to NI data

Currently the names are on the Memberships; these should be on the Person.

Make the `source` for PopIt data the website, not the API

Directing people to the API is a little unfriendly.

Have a separate Morph-pull account

Morph now publicly shows which accounts have downloaded data over their API. Have a distinct EveryPolitician account for pulling.

Store seat-count for each Legislature/Chamber

Even if we don't know how to express it in Popolo yet, we should gather it.

Tanzania: Fetch individual Member details (before October)

The current Members have pages with more information, e.g. http://www.parliament.go.tz/index.php/members/mpcvs/1603/4

Store government make-up per Term/Legislature

Even if we don't know how to express it in Popolo yet, we should gather it.

Turkey: Get older data from Turkish Wikipedia

https://tr.wikipedia.org/wiki/T%C3%BCrkiye_B%C3%BCy%C3%BCk_Millet_Meclisi#Yasama_d.C3.B6nemleri
https://tr.wikipedia.org/wiki/TBMM_20._d%C3%B6nem_milletvekilleri_listesi etc.

Niger

http://www.assemblee.ne/index.php/organes/les-deputes/161-la-liste-des-deputes

Monaco

Looks easy enough: http://www.conseil-national.mc/index.php/les-elus/les-groupes-politiques

Spain

http://proyectocolibri.es/documentacion/

Mexico

• 62nd: http://sitl.diputados.gob.mx/LXII_leg/listado_diputados_gpnp.php?tipot=TOTAL
• 61st: http://sitl.diputados.gob.mx/LXI_leg/listado_diputados_gpnp.php?tipot=TOTAL
• 60th: http://sitl.diputados.gob.mx/listado_diputados_gpnp.php?tipot=TOTAL
• Wikipedia: https://en.wikipedia.org/wiki/LXII_Legislature_of_the_Mexican_Congress + lots of historical

Ukraine

http://w1.c1.rada.gov.ua/pls/site2/p_deputat_list

includes 7 previous convocations, back to 1990

Add `date_added` to the Meta information

as a precursor towards everypolitician/viewer-sinatra#153

Tanzania

http://www.parliament.go.tz/index.php/members/memberslist/all/all/4

3 terms, looks easy to scrape

Ensure all People and Organizations have names

Trinidad and Tobago

There is an election coming up for the House of Representatives. Try to capture the old data before it's too late (if it's not already).

New Zealand

official: http://www.parliament.nz/en-nz/mpp/mps/current + http://www.parliament.nz/en-nz/mpp/mps/former

current: http://votes.wotfun.com/parliament/

historic: https://en.wikipedia.org/wiki/47th_New_Zealand_Parliament etc

Brazil: Get more precise Term dates

Currently they're only at the granularity of years — try to find a source for actual dates.

Add a locale code to all data

France

http://cpc.regardscitoyens.org/trac/wiki/API

Getting current data is easy. There appears to be a call for previous terms, but you need to know the value to give.

Brazil: Get list of parties

Currently they're just abbreviations.

Possible source: https://en.wikipedia.org/wiki/55th_Legislature_of_the_National_Congress

Switzerland

Official: http://www.parlament.ch/E/ORGANE-MITGLIEDER/NATIONALRAT/MITGLIEDER-NR-A-Z/Pages/default.aspx

Scotland

Can presumably copy more of the Northern Ireland code

Madagascar

http://www.assemblee-nationale.mg/?page_id=5217

Suriname (election due May 25th)

http://www.dna.sr/het-politiek-college/leden/

Ensure the JSON stays in sorted order

JSON.pretty_generate doesn't seem to have an equivalent to sort_keys=True

Shelling out to python just for that seems overkill, so perhaps something like http://stackoverflow.com/a/29612016 might do?

UK

https://github.com/mysociety/parlparse/tree/master/members

mysociety/parlparse#23

Burundi

Burundi has an election on May 26th, so it would be useful to scrape the existing data before then in case it disappears afterwards.

Find a better `source` for Parldata sites

Directing people to the API is a little unfriendly.

Angola

http://www.parlamento.ao/web/guest/deputados-e-grupos-parlamentares/deputados/lista#http://www.parlamento.ao/glue/AN_Navigation.jsp?

Scraping the site itself is easy (and there doesn't appear to be any extra information on the Person page that's not on the List page) — but the next/prev links are JS driven, so we'll probably need to use Morph's Phantom.js support

Chile: Lots of historic data

• http://historiapolitica.bcn.cl/resenas_parlamentarias
• http://www.camara.cl/camara/galeria4145.aspx

(from @lfalvarez)

Angola: Scrape individual MP pages

e.g. http://www.parlamento.ao/web/abilio.numa

Rwanda

http://www.parliament.gov.rw/chamber-of-deputies/members-profile/chamber-of-deputies-members-profile/

Also historic information at http://www.parliament.gov.rw/chamber-of-deputies/members-profile/former-chamber-of-deputies/ — but doesn't seem to say when they were members.

DRC

http://www.assemblee-nationale.cd/v2/?page_id=4087

Brazil: Get coalition information over time.

This is a case where it's essential to group the parties!

e.g. https://en.wikipedia.org/wiki/55th_Legislature_of_the_National_Congress