Delayed data - last update 4 days ago about covid-19

[anuveyatsu]

hey @snympi thanks for reporting. I can confirm that we are working on fixing this so it will be updated ASAP.

[Morisset]

It is possible to generate the tables locally on your own computer by running the following script:
https://raw.githubusercontent.com/datasets/covid-19/master/scripts/process.py
I did a fork of the repository and run the process script in it, then commit the new files, so on my fork the data are uptodate. I will try to run it everyday.
My fork is there: https://github.com/Morisset/covid-19

[Morisset]

OK, I'll try to do it. I created a crontab that will to the job everyday at midnight PDT: run process.py, commit to git, push to my fork on github. But I do not know how to request the pull from a shell script. Any tip on this @sglavoie ?

[snympi]

@anuveyatsu thanks for confirming. Any ETA on the fix?

[lordruzi]

Hi everyone,
Any update on this?

[mattlgardner]

Hi all - Guessing still no change on the missing data?

[raniermc]

Hey, this project it's over?

[olwal]

Same here, any ETA on bringing back the up-to-date data? (I see that it is being worked on in #93

Would be good to know before people start migrating to different data sources.
I really appreciated your work and how easy it was to integrate this w/ for example the population datasets.
It also seems that there's a few people that might be willing to help (me included). Let us know!

Either way, thanks for your great work!

[sdangt]

This is really too bad. I think a lot of people were using this data, which was presented in a nice clean format. It would just be helpful to know in advance if a project is going to stop. I incorporated this into my spreadsheet analysis, but the data here is now nearly two weeks out of date, and there are no updates about it, so it seems like people just disappeared into the night. Thanks for the effort initially. No one knew for sure how long this pandemic would go, but I think it was clear that it would probably still be troublesome at this point, and it probably will be into next year as well. So data needs to go at least through that period if it is to be most useful.

[sglavoie]

Thank you @Morisset for sharing this information with others and for hosting a backup of the latest data 👍. This is indeed a viable solution while we are working on a fix. I have updated the data manually in this repo for now in 3497aad.

[Morisset]

Great @sglavoie ! How do we proceed now? Do I update the data manually on my fork everyday, or will you do it?

[sglavoie]

If you would like to give us a hand with this while we get to provide a more permanent solution on our side, that would be quite useful! You might want to send a daily pull request while we get this in order so you are recognized as a valuable contributor in the process and we can all enjoy up-to-date data 😉.

[sglavoie]

@Morisset: this could be useful for the PR. If not, don't hesitate to let us know when you're updating. Else, I will have a look and merge the changes a bit later. In any case, thanks for your helping hand! Hopefully we'll get things in order soon enough 😉.

[Morisset]

OK, I'll have the following script running every midnight PDT:
#!/bin/tcsh set TODAY=date +"%m-%d-%y"cd /home/morisset/covid-19 python scripts/process.py git commit -a -m "Update $TODAY" git push hub pull-request -m "Test command line" -b datasets:master -h Morisset:master
I just ran it, data are note updated, but one json file was. The script run correctly, a request have been sent, let's see tonight if OK from crontab.

[jochym]

@sglavoie Let me ask: why the procedure takes so long? Can we do something about that? Can I help in some way? Maybe we should rethink the algorithm - otherwise I think that this may be unsustainable i the coming months (this thing is not going away soon, sedly).

[sglavoie]

@Morisset: Thank you, changes merged! However, could you manually pull the updates before your script runs again so we are in sync and can merge cleanly? That would be great. Thanks for helping by the way 😉.

@jochym: we haven't been able to work on the issue so far, but thankfully we'll start bringing a fix really soon, don't you worry! 😄 We will keep the data updated for now with the kind help from @Morisset, which is mostly an automated process. The script is being run off the same code we have and produces the expected output, but we will optimize the code so it doesn't take so long to get the changes in. A partial fix already exists using a different library (Pandas): I will let you know once it's done and we have QA the results before making those changes to the repo. 👍

[sglavoie]

FIXED: Since commit c5b86f8 from 3 days ago, the data has been successfully updating on GitHub. Closing this issue as nothing else needs to be done for now with the code itself.

Note: There is a problem regarding the deployment to DataHub, but this is unrelated to the data itself, which is correctly being updated here.