linz / download_ta_bdys Goto Github PK

Add an option to create the db schema and schema objects, either as empty or do an import straight away.

additional index needed on territorial_authority_grid table

create index territorial_authority_grid_territorial_authority_ogc_fid_idx on admin_bdys.territorial_authority_grid(territorial_authority_ogc_fid);

https://github.com/linz/AIMS/issues/60#event-655254329

@josephramsay. As discussed, once you've incorporated the fix from the table_version function and got it working again. We will do a code review with @imincik and discuss any improvements that could be made.

Will need to notify [email protected] and [email protected] when this is run with suitable information. Will need to make it clear whether processing was successful or not.

Use the data's natural primary key (the TA code) rather than generating an ogc_fid key during import.
This will be needed for new AIMS requirements.

AIMS download_admin_bdys Design.docx

See this issue: https://github.com/linz/LI-tasks/issues/30

Needs to be rewritten to also import meshblocks and NZFS boundaries
These may include FTP or CIFS sources.

Data has to be filtered/mapped to more useable field names.

Details can be found here:
https://github.com/billgeo/AIMS-reference-data/blob/master/import_localities.sh
https://github.com/billgeo/AIMS-reference-data/blob/master/import_meshblocks.sh

Would be good to make this more configurable rather than hardcoded and also use YAML for configuration.

Things like:

1. Data sources
2. Source -> Target schema mapping
3. Logging email details

Good news, @josephramsay, can mount the meshblock folder as a CIFS mount.

See the Wiki:
http://geodeticwiki/Meshblocks

Addressing request to provide a means of user validation pausing processing before final import.
Initial implementation.

1. DAB split into two parts; (1) File download and DB import. (2) Table mapping and versioning
2. Notification to users following completion of Part-1 @LinzDSie
3. Web interface (on common tomcat installation) taking user input (Y/N) to trigger Part-2

Additionally, provide changeset information in notification and/or web interface, also validation functions..

Capturing this as a separate issue. Need to add basic validation rules for admin data imports.

Final step in the process is to run the AIMS database functions for updating reference data. Will need to check the return codes and raise message appropriately.

fnreferencedataupdatelocality
fnreferencedataupdatemeshblock

Move the logic to create and maintain the territorial_authority_grid table from the BDE processor to here. See https://github.com/linz/linz_bde_uploader/blob/824f9fad7aafd9538776d06218d602b398d207d8/CHANGELOG.md

meshblock fetching is done over sftp using the python paramiko library that only works on newer linux installations

This seems to be hardcoded to get 2013 boundaries? Stats now has a new endpoint where they add a new service every year. Could scan these and extract the latest.

http://maps.stats.govt.nz/wss/service/arcgis1/guest/Boundaries/Local_government/MapServer/13

Data should be imported incrementally, i.e. only import changes to the data based on primary key (don't truncate and insert).

Use the functions from the table versioning extensions to create a diff and then apply it. Would suggest importing to a temp table first.

ver_get_table_differences
ver_apply_table_differences