n0rdlicht / rki-vaccination-scraper Goto Github PK

Publish a document showing when which key was added / deprecated and hence has data associated with it.

Hi @n0rdlicht,

Goodtables.io is going to be deprecated in 2022, we, therefore, recommend migrating to the new Frictionless Repository (https://repository.frictionlessdata.io/) continuous data validation system provided by Frictionless Data. The core difference between the two projects is that Frictionless Repository doesn't rely on any hosted infrastructure except for Github Actions which makes this project more sustainable. Also, it uses a newer Frictionless Framework under the hood that brought many improvements over the old goodtables-py library in terms of validation quality and performance.

As usual, if you have any doubts or questions, please come and ask in our Discord chat or in the GitHub Discussion.

but sum_booster_moderna and sum_booster_biontech are available

• Set-up auto publishing to Datahub.
• Add basic chart

Some values are not yet in de-vaccinations due to missing keys. Backfilling will be done from git history.

Improve data cleaning to avoid data pollution in de-vaccinations-current.csv and de-vaccinations.csv by RKI annotations and data structure changes.

E.g. see 2c18d5c#diff-198151b974d013a0fe8c933a27b95090ab4f92ec9a7032b58c8071d292e73a10R427

Publish CSVs / DataPackage as Datasette.

• no more indications "Pflege", "medizinisch", "Alter", "Beruf"
• age <60 and 60+
• added vaccinations in doctor's practices

Due to completly new structure of the underlying RKI excel file, the current pipeline is broken.

• Now two relevant tabs instead of one, one for sums and one for different indications
• New: Daily numbers for initial and booster vaccinations
• New: Daily numbers for different vaccine types (currently BioNTech and Moderna)

To future proof and simplify the code should be migrated to frictionless-py Pipelines/Transformations as the API is already using it this would greatly simplify dependencies.

e.g.
Germany,DE,nation,sum_initial_moderna,3597780,83166711,7.260192121821434,2021-04-26
while on 2021-04-25:
Germany,DE,nation,sum_initial_moderna,1069222,83166711,7.166621029416445,2021-04-25
was 2.5 million less...
rki excel reports 1099371 on 2021-04-26