Use DTM, not PLUTO about labs-tax-lot-selector

[allthesignals]

Could set up a cron job in a dokku container that syncs large files through ogr and Carto's endpoint: https://carto.com/blog/data-sync-ogr

[chriswhong]

The trick is that the postGIS database is internal to the docker container that carto is running in, so you don't have the luxury of just doing regular database commands. The import API, or the GUI are the only ways in, and large files don't play well...

There are some workarounds where we can expose the database to the host machine... this would be worth looking into, but then you have to manually cartodbfytable() and stuff.

[chriswhong]

Actually, just downloaded the DTM... the polygons layer alone, zipped is 136MB, which might be manageable to send into the Carto import API.

[allthesignals]

I like the idea of exposing the internal database, or, even better, having that database running outside of the container on AWS or a Droplet (if it's ever rebuilt, all the data would be lost, no?).

Separately, maybe we just have a separate dokku app with a number of shell scripts triggered by crontab. Each script can bulk sync special cases like this. In this case, the script might split the file into manageable pieces using ogr2ogr.

[chriswhong]

Yes, it would be ideal to have the DB in a separate container altogether. We would need to dedicate some time to getting this set up in production... it provides for both direct access to the DB for etl, and a more consistent upgrade path for the applications.