everypolitician / commons-explorer

(Currently) hacky Sinatra app using commons-builder to show what's in Wikidata

EveryPolitician

Data about every national legislature in the world, freely available for you to use

• everypolitician.org — data | about
• Report an issue

Repo summary

These are some of the key repos in the EveryPolitician family. There are others.

• everypolitician (this repo): contains no code, but is where issues/tickets for the whole project live

• everypolitician-data: where the data is stored -- but if you want to download it, get it from:

  • human? go via the EveryPolitician website
  • program? use the RawGit CDN, via links in countries.json, which we explain here

• viewer-static: the live website http://everypolitician.org (gh-pages)

• viewer-sinatra: Sinatra app for generating a dynamic version EveryPolitician website

• webhook-manager: sends out EveryPolitician WebHooks: register your URL here!

• everypolitician-docs: documentation at http://docs.everypolitician.org/ (gh-pages)

• rebuilder rebuilds data from source

• libraries for easily manipulating EveryPolitician data (useful for all devs, but we use the Ruby ones ourselves, of course!):

  • Ruby: everypolitician-ruby and everypolitician-popolo.
  • Python: everypolitician and everypolitician-popolo

• handy gems we use when getting the data: wikidata-fetcher, wikisnakker, twitter_username_extractor, facebook_username_extractor, twitter_list, scraped_page_archive

• gender-balance: repo for the Gender Balance website that crowdsources gender data for EveryPolitician

• data_pr_change_summarizer: code used by the bot to review a data PR and leave a helpful summary as a comment

The repos for many of our scrapers are kept separately in github.com/everypolitician-scrapers.

Technical blog

The EveryPolitician bot's own page is a good jumping-off point to lots of semi-technical explanations of what's going on (it has its own blog on Medium). For example:

• how the website is built (spoiler: viewer-sinatra → viewer-static)
• how webhooks are used (you can easily register your app!)
• how the scrapers run (many live on morph.io)

The bot is on twitter as @everypolitician

Contributing

If you have data for us, or know where to get it, please read our page about how to contribute.

Team

EveryPolitician is a mySociety project.

commons-explorer's Issues

Better distinguish that legislatures and executives belong to geographical regions.

At the moment, legislatures and executives are grouped into those two categories. In actuality, there is an additional grouping of geographical region, which can potentially include multiple legislatures.

For example, https://commons-explorer.herokuapp.com/country/proto-commons-canada shows a large mix of both types. In actuality, the Country consists of a National level with a bicameral legislature and multiple Regions with unicameral legislatures.

This should be more readily apparent.

Link to relevant Wikidata items

Items have a relevant entity in Wikidata - pages should offer links to these wherever it would be intuitive to expect to be able to visit the entity directly.

Legislature pages should include summary statistics

A page for a legislature should include summary statistics such as:

• Expected number of members
• Total number of members and member_ships_ (one person may have multiple)
• Number of current members/memberships
• Number of ended members/memberships

Show number of seats vs current number of members for a given legislature

For any given legislature, highlight the number of seats in the legislature and the number of members actually returned by our current query.

Some form of high-level visibility of this (percentage, absolute number of difference, progress bar etc) will make it easier to identify how 'complete' a legislature is likely to be.

Provide list of known exceptions for any given country or legislature

NB: This is a grand vision ticket which describes an eventual goal to improve overall usefulness, not something which should be considered an essential feature.

Commons Explorer should be able to identify issues which are known to be incorrect, impossible or inconsistent, and highlight them for attention.

Possible examples include:

• A legislature does not have a number of seats.
• The number of members in a legislature does not match the number of seats.
• In a term-based legislature, the most recent term appears to have ended in the past.

In general there is no way to programmatically state if the contents of the commons are correct, but it should be possible to highlight cases where we know for a fact that they are not.

Include WDQS links to the queries used to generate any particular page.

At present, every page in the Explorer is generated from a specific query in the proto-commons repo. It would make sense for there to be a direct link to query.wikidata.org for any page which automatically includes the query used, making it easier to use the Explorer as a starting point for further investigation.

Cache query results

Queries take a significant amount of time to run; query results should be cached in some way, either on first request for a page or by processing a batch job of all queries at a given interval.

It must also be possible to manually refresh a query.

Make it clear for each item on a country page which membership entity is causing it to appear

eg https://commons-explorer.herokuapp.com/country/proto-commons-pakistan shows the Senate twice, for two different positions held (member and chairman).

Ideally this should either only appear once (with subsections on the subsequent page to reflect different membership types), or each appearance should detail which positions it is listing.

Legislature data should include method used to determine current member

At present we use three distinct methods to find people who should be considered as current members of any given legislature.

• Generic "Member of X" position.
• "Member of X" with an "electoral term" qualifier.
• "Member of the nth term of X".

It would be useful to explicitly declare somewhere which kind of modelling was being used. This may help improve queries being built, or allow for easier exception reporting in future.

Better represent information regarding a geographical region

The concept of a 'region' is currently poorly represented in general, but it would be worthwhile to be able to explore information which directly pertains to a region, as opposed to a concept within it, such as:

• Head of government
• Executive body
• Population
• Legislative body

This will also help in automated sanity checking, able to throw exceptions where (eg):

• A region does not have a head of government
• The executive given for a region does not appear in a query
• Legislatures within a region are not the (or part of) the legislative body of the region

Separate pages for executive and legislative

Or AJAX some of the queries, or split by admin area. This might negate the need to do #9.

Limit or change the way we ask for labels

To help speed up page loading @alexsdutton suggests we can limit/change the labels we ask for…

A list of members of a legislature should make it clear who is a current member

The pages listing members of a legislature should make it visually apparent who is a current member vs who is a past member.

A possible edge case is where someone has left and rejoined within a session.

Executive pages currently error

Pages for a region's executive, eg https://commons-explorer.herokuapp.com/executive/proto-commons-united-kingdom/Q112014&position_ids=Q14211, currently cause a server error.