simonw / cdc-vaccination-history

A git scraper recording the CDC's Covid Data Tracker numbers on number of vaccinations per state.

cdc-vaccination-history

Project retired as of 25th October 2023

A git scraper recording the CDC's Covid Data Tracker numbers on number of vaccinations per state.

Archives the JSON from https://covid.cdc.gov/covid-data-tracker/COVIDData/getAjaxData?id=vaccination_data every time it changes, checking three times an hour.

Watch Git scraping, the five minute lightning talk to see me live-code the creation of this repository.

This data as CSV

If you want to grab the entire dataset I'm now publishing it as two CSV files here:

• https://cdc-vaccination-history-csv.datasette.io/daily_reports.csv - ~5.5MB
• https://cdc-vaccination-history-csv.datasette.io/daily_reports_counties.csv - ~90MB

This data in Datasette

The build_database.py script loops through the full commit history and uses it to build a SQLite database with a row for every daily report, mainly as a demonstration of how Python code can be used to extract data from a git scraped repository.

That database is then deployed using Datasette - you can browse the data at https://cdc-vaccination-history.datasette.io/cdc/daily_reports

You can filter down to individual states like so:

• https://cdc-vaccination-history.datasette.io/cdc/daily_reports?_sort=id&Location__exact=CA

Take a look at the scrape.yml GitHub Actions workflow to see how the scraper runs, and how the data is then built into a database and published to Vercel using datasette publish.

Should you trust these numbers?

I honestly don't know. These are not coming from a documented API - I found it using the Firefox developer tools network pane. I don't know how the CDC are sourcing these. I don't know if they themselves consider them to be accurate.

All I know is that these are the numbers they are displaying on their own site - so you should treat this repository as tracking "numbers that were displayed on the CDC's website" as opposed to assuming it represents the full truth on the ground.

cdc-vaccination-history's Issues

Run less frequently

3 times an hour is unnecessary - the commit log at https://github.com/simonw/cdc-vaccination-history/commits/main shows a maximum of three updates a day, usually just one or two.

Old county JSONs

Hi, do you archive the prior JSONs for the county vaccination data? I only see the latest here. Thanks!

Speed up the database build step

It's taking 22 minutes to build the SQLite database now!

Move from Vercel to Cloud Run

Deploys to Vercel started failing once the DB file exceeded about 100MB.

Publish CSV versions of the data as well

Originally posted by @simonw in #4 (comment)

Also inspired by PR #5 by @alexose

How do you download the full data set?

I am using the link provided: https://cdc-vaccination-history.datasette.io/cdc/daily_reports_counties
Under CSV options I check off "download file" and "stream all rows", and then click on "export csv". I get the following error:

An error occurred with your deployment

504: GATEWAY_TIMEOUT Code: FUNCTION_INVOCATION_TIMEOUT ID: iad1::xd25w-1621449834980-16b0300f009e

If you are a visitor, contact the website owner or try again later.
If you are the owner, check the logs for the application error. 

Is there another way you could just post the whole file? Like maybe a separate git hub repository could hold the results?

Build database and deploy to Vercel

So people can actually see the data that has been collected.

Scrape county information too

Available at https://covid.cdc.gov/covid-data-tracker/COVIDData/getAjaxData?id=vaccination_county_condensed_data