Write scheduled GH actions to pull country epi data about covasim

[gwincr11]

When you say cache it locally, do you actually mean check it in with a pull request?

[inc0]

Or directly commit into repo, at least for now, via deploy key. @cliffckerr are you ok with this approach?

Every day (or hour) we could pull latest data from JHL, process it and cache it locally. We'd need to create deploy key and save it as action secret

[cliffckerr]

@inc0 if we're committing data, this should probably not be in the covasim repo -- potentially synthpops but maybe best to have a separate module/repo for it?

[inc0]

or actually we can just have something like python download_data.py and entry to .gitignore?

[ckerr-IDM]

works for me! we can have a gitignored data folder and a script to update it

[gwincr11]

I am working on a script to pull in the neherlab data as part of: #45

I ended up with a folder structure of:
covasim/datasets/sources/ Where the data will live
covasim/dataset/translators.py Which is holding the translators to get datasets to the format we want
covasim/datasets/update_sources.py Which pull in the source files when run.

I was also thinking about adding a reference file that could tell us which of the source files contain which countries data. This could be generated by update sources and dropped in the sources file.

I was speaking with Scott Ayers and he could also use the reference file for some front end work he is doing. So it serves a few purposes.

[hamelsmu]

Some comments/questions

• The John Hopkins data is only updated daily, no need to pull more often
• Why do we need to check in a copy of the data into this repo? We can reference the URL of the data directly. Most data science libraries allow you to reference the URLs of CSV files as a filehandle

[ckerr-IDM]

@hamelsmu yes, i think we're in agreement -- we'd have a script to manually update the data into a git-ignored folder

[hamelsmu]

Ah yes, sorry I misread the thread -- my apologies