codespell --ignore-words-list="humber,ons" --quiet-level=2

./maven/datasets/coronavirus/README.md:7: authorative ==> authoritative
./maven/datasets/coronavirus/README.md:7: authorative ==> authoritative
./maven/datasets/general_election/README.md:7: authorative ==> authoritative
./maven/datasets/general_election/README.md:7: authorative ==> authoritative

Should be similar to #19

Do as the thing says. Cover both get.py and existing pipelines.

Focus for 0.1 is to get it to a releasable state where I'm not entirely ashamed of how basic it all is. Task priority is on the Core Projects board.

Tasks

Core

• #3 is done and 0.0.3 is live!
• #9: Add some tests
• #10: Add test info to setup.py

Pipeline design

• Add dataset caching (design inspiration)

New pipelines

• #19: Clean up 2015 results to use HoC data
• #20: Add in 2017 UK/GB election results data
• #21: Fix up UK2015Model to work with updated HoC data
• #22: Create UK2017Model
• #29: Update polling data using Mark Pack's PollBase
• Add in this notebook in order to make it easy to get to a model ready dataset.

Holding pen for things I'd like to look at integrating but aren't quite ready to go into a release yet.

Ideas

• Consider using Poetry/Hatch to smooth deploy process
• Integrate with https://dataverse.org/
• Bring in data from https://www.pippanorris.com/data

Consider adding CLI functionality

The currently processed 2015 results data isn't even that nice. Integrate further cleaning and ID mapping from the SixFifty model.

This currently uses the Electoral Commission's results dataset. Earlier this year, they stopped hosting results and this is now officially published by the House of Commons library at parliament.uk. We need to bring this code up-to-date so it uses the new data (which is in a different format!) for the 2015 results: https://researchbriefings.parliament.uk/ResearchBriefing/Summary/CBP-7186

The goal is the same, we want to turn this into a dataset with the same format as the current code with columns something like:

Press Association ID Number    Constituency ID    Constituency Name    Constituency Type    County    Region ID    Region    Country    Election Year    Electorate    Valid Votes    con    lab    ld    ukip    grn    snp    pc    other    con_pc    lab_pc    ld_pc    ukip_pc    grn_pc    snp_pc    pc_pc    other_pc    geo    winner

The new dataset also switches to using just ons_id (which is the same as Constituency ID in the previous cleaned results) and not Press Association ID Number, this is fine, you can drop that column.

Add test info to setup.py

gov.uk

• COVID-19: track coronavirus cases
  • Desktop dashboard
  • Mobile dashboard
  • daily indicators
    • DailyIndicators.xlsx data
    • Date, total uk cases, new uk cases, total uk deaths, england/scotland/wales/NI cases
    • 15th Mar only, no history
  • number of daily confirmed cases
    • DailyConfirmedCases.xlsx data
    • date, CMO new cases, cumulative cases
    • 31st Jan to 15th Mar history
  • NHS England Regions cases table
    • NHSR_Cases_table.csv data
    • GSS region (england only, east/london/midlands/etc), region name, total cases
    • latest only, no history
  • UTLA cases table
    • CountyUAs_cases_table.csv data
    • By UTLA (upper tier local authority) e.g. Barnet, Bristol, etc
    • id, name, total cases
    • local only, no history
• Deprecated
  • Table of confirmed cases of COVID-19 in England

Individual/historical aggregators

• https://github.com/tomwhite/covid-19-uk-data
  • historical tests, cases, deaths for UK; and breakouts with some time series; for england, wales, scotland, NI
• https://github.com/emmadoughty/Daily_COVID-19
  • similar to tomwhite, possibly less coverage?

We changed some things (e.g. removing PANO in favour of ONS codes). The UK2015Model is also not idiomatic pandas in many places so a good opportunity to clean it up.

Add docs

Similar to UK2015Model, I have a notebook I can share which implements this.

Design inspiration: gensim's downloader functions

• https://rare-technologies.com/new-download-api-for-pretrained-nlp-models-and-datasets-in-gensim/
• https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/downloader.py#L291
• https://github.com/RaRe-Technologies/gensim-data/blob/master/list.json

https://python-packaging.readthedocs.io/

https://www.markpack.org.uk/opinion-polls/

Tasks

• #2: Run through Python packaging to create installable package.
• Decide on a license.
• Push Maven to PyPI as a 0.0.1 release.
• Add a minimally useful pipeline.

Also add additionally recommended steps from https://python-packaging.readthedocs.io/

https://github.com/CSSEGISandData/COVID-19/

Focus for 0.2 will be to bring in some more Python best practices. Feeling quite inspired after catching up with Cheuk, there's a lot of things going on in picknmix that are really nice.

Tasks

• #31: Add 2019 UK GE results
• #12: Add docs
• #11: Consider adding CLI functionality
• Use Cookiecutter & cookiecutter-pypackage.
• Add CI.
• Add badges for pypi version, build and docs status. (And more?)
• Use http://editorconfig.org
• Add AUTHORS
• Add CONTRIBUTING
• Add Makefile
• Add requirements.txt as well as Pipfiles (stick with Pipenv or switch back to pip-tools?)
• Add setup.cfg
• Run tests with tox if haven't done so yet
• Version datasets and integrate this into caching spec.