balazsdukai / batch3dfier

A wrapper around 3dfier to 3dfy datasets in batch.

batch3dfier

local

Description

This Python program helps you to create 3D models of LoD1 from massive data sets. The idea for the program came from the need to efficiently create a 3D model for the whole Netherlands, using building footprints from the BAG dataset and the AHN pointcloud.

The software 3dfier is the heart of batch3dfier, where batch3dfier adds a tiling framework, allowing to scale to an arbitrary size of input data set.

As of version 0.7.0 batch3dfier:

• Can 3dfy 2D footprints (polygons) that are on a single layer, in this case, a single database table (e.g. building footprints).
• Its footprints module helps in partitioning a large footprints data set into tiles. Only rectangular tiles are handled.
• 3dfies the footprints tile per tile, running concurrent threads. And as input it accepts,
  • a polygon of the desired extent,
  • a list of footprint tile IDs,
  • all to process all tiles in the footprint tile index.
• Expects that the footprints, the tile indexes of footprints and pointcloud are stored in a PostgreSQL database, the pointcloud stored in files.
• In case of CSV-BUILDINGS-MULTIPLE output format, it can copy the CSV contents into postgres, thus the different buildings heights can be joined on the footprints.
• Has been only tested with BAG and AHN data sets.
• It does not mirror the complete feature set of 3dfier. Apart from those listed above, the most notable differences are:
  • 3dfied tiles are not stitched together to create a 'watertight' model.
  • Some of the 3dfier configuration parameters are not included in the batch3dfier configuration file. Mainly because there was no need for them. If you notice that something is missing, let me know in an issue or submit a pull request.

To get further information please refer to the documentation.

Plans for version 1.0.0

See the related issue.

Requirements

3dfier

Python 3.4+

PostgreSQL 9.6+

The package has been tested with Python3.4-3.6 on Linux with the following packages:

• PyYAML (3.11)
• psycopg2 (2.7)
• Fiona (1.7.1)
• Shapely (1.5.17)

Install and run

• Download and install the latest release from GitHub:

  $ pip3 install git+https://github.com/balazsdukai/batch3dfier

• Alternatively, install from the wheel in /dist.

• Run batch3dfier from the command line:

  $ batch3dfy ./batch3dfier_config.yml

  Where batch3dfier_config.yml is the YAML configuration file that batch3dfier uses (similarly to 3dfier).

• Get help:

  $ batch3dfy -h

• In order to process several tiles efficiently batch3dfier starts 3 concurrent threads by default, each of them processing a single tile at a time. Set the number of threads:

  $ batch3dfy -t 4 ./batch3dfier_config.yml

• In case the output format CSV-BUILDING-MULTIPLE is used, it is possible to copy the CSV files into a posgres table. The default is to keep the CSV files after import. To delete the files after import:

  $ batch3dfy --del-csv ./batch3dfier_config.yml

• Run the tests:

  $ pytest batch3dfier/tests

Example data sets are in batch3dfier/example_data.

batch3dfier's Issues

implement DB connection pooling

http://initd.org/psycopg/docs/pool.html#psycopg2.pool.ThreadedConnectionPool

Setup installs version 0.5.0

In setup.py it states version='0.5.0' instead of version='0.6.0'

v1.0.0

• dict input for the footprints module as well (as in config)
• create footprints command line tool. Probably will require a (separate) yaml config.
• handle multiple footprint layers (e.g. building, road, water etc) – BGT
• stich the output into a watertight model – BGT
• grid module for generating tile indexes
• automate the whole workflow of grid + footprints (possibly into a single function call)
• improve reporting so that it also includes failures from 3dfier

remove pand_centroid table after bag3d is created

because bag3d is materialized, no need to keep pand_centroid

tile_index:polygons and tile_index:elevation must be different tables

The footprint and point cloud tile indices must be in a separate table, thus not possible to use the point cloud tile index as footprint index. In case the same tile index is used for both pc and footprints, it needs to be imported twice, into separate tables into postgres.

Such as:

tile_index:
    polygons:
        schema: tile_index
        table: ahn_index
    elevation: 
        schema: tile_index
        table: ahn_index

This is because find_pc_tiles() references both indexes:

                        SELECT
                            {table_pc}.{field_pc_unit}
                        FROM
                            {schema_pc}.{table_pc},
                            {schema_ftpr}.{table_ftpr}
                        WHERE
                            {table_ftpr}.{field_ftpr_unit} = {tile}
                            AND st_intersects(
                                {table_pc}.{field_pc_geom},
                                {table_ftpr}.{field_ftpr_geom}
                            );

The traceback indicates this problem:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/home/bdukai/.local/lib/python3.5/site-packages/batch3dfier/batch3dfierapp.py", line 208, in run
    process_data(self.name, self.q)
  File "/home/bdukai/.local/lib/python3.5/site-packages/batch3dfier/batch3dfierapp.py", line 238, in process_data
    thread=threadName)
  File "/home/bdukai/.local/lib/python3.5/site-packages/batch3dfier/config.py", line 60, in call_3dfier
    prefix_tile_footprint=prefix_tile_footprint)
  File "/home/bdukai/.local/lib/python3.5/site-packages/batch3dfier/config.py", line 257, in find_pc_tiles
    resultset = db.getQuery(query)
  File "/home/bdukai/.local/lib/python3.5/site-packages/batch3dfier/db.py", line 59, in getQuery
    cur.execute(query)
psycopg2.ProgrammingError: table name "ahn_index" specified more than once

safe passwd handling

be able to use .pgpass instead of giving passwords in cleartext

postgres bottleneck?

batch3dfier reads the footprints from postgres views and opens a single connection for all the operations. Apparently prior version 9.6, postgres can only use one CPU thread/core at a time. I'm not sure about it, but this might mean that regardless of how concurrency is implemented, the bottleneck might become the I/O from postgres as it can only run on a single thread (because there is a single open connection). It seems like that this problem could be side-stepped by opening a separate postgres connection for each thread.

sanitize logging

Sanitize log file creation from the individual modules. Take care that the log file size won't explode due to 3dfier messages.

• Everything should go into a single log file.
• Also, probably there shouldn't be a need for a cron.log.
• make 3dfier silent, because it pollutes the log quite a bit

config.pc_name_dict expects a list

should also work when input_elevation:dataset_name is a string