owid / owid-datasette Goto Github PK

metadata.yml contains metadata about tables for Datasette. It lives in this repository, but those tables live in analytics repository. We could move the file there to be closer to the tables and then fetch it from here (see update.sh), but then it would be further from the actual Datasette instance. It's not yet clear how this should work.

See this #23

Just opening this issue as a way to keep track of what I just found, in a search through the Datasette plugin page:

We probably want to enable auth soon, so we can sync data tables like pageviews over without exposing them.
For this, these three plugins sound super useful:

• datasette-auth-github, where we can use GH as a auth provider, and grant access to all users that are part of the @owid organization
• datasette-public, where we can specify a bunch of tables to be publicly accessible, without the need to log in
• datasette-saved-queries, where users can save queries in their account

Currently, there is a (broken) cronjob on the analytics server that is supposed to build the sqlite file and then does a dummy git commit to owid-datasette and pushes this. We want to get rid of analytics soon and so this should instead be done in the datasette LXC container instead.

Instead of doing dummy commits to trigger redeploys, the digital ocean API should be called to trigger a restart of the owid-public container after the sqlite file has been uploaded.

When this is done, the datasette folder on analytics and the cronjob can be deleted

This is probably a minor thing, but in case there are blocks in the content (references with <!-- wp:block {"ref":24542} /-->) like in https://ourworldindata.org/world-population-growth#population-growth, then those charts won't be in post_charts table. To fix this we'd have to lookup all references and materialize them in content before searching for links.
(This has been fixed in analytics repo)

Hey there!

Was trying to play with the public SQLite database of OWID and couldn't find it.

I've tried these two URIs:

• s3://owid-public/owid-public.sqlite.gz
• s3://owid-public/owid.db.gz

Something like https://datasette.io/plugins/sqlite-regex would allow us to do a couple of useful analytic aggregation queries on the private instance.

(Specifically extracting references to particular DoDs by ID and then aggregating how many pageviews each DoD has.)