sdfidk / malstroem

Tools for screening of bluespots and water flow between bluespots

malstroem

Tools for screening of bluespots and water flow between bluespots

Malstrøm is based on the ideas presented by Balstrøm and Crawford (2018).

Balstrøm, T., Crawford, D., 2018, Arc-Malstrøm: A 1D hydrologic screening method for stormwater assessments based on geometric networks, Computers & Geosciences, vol. 116, pp 64-73, doi:10.1016/j.cageo.2018.04.010

Features

malstroem provides command line tools and a python api to calculate:

• Depressionless (filled, hydrologically conditioned) terrain models
• Surface flow directions
• Accumulated flow
• Blue spots
• Local watershed for each bluespot
• Pour points (point where water leaves blue spot when it is filled to its maximum capacity)
• Flow paths between blue spots
• Fill volume at specified rain incidents
• Spill over between blue spots at specified rain incidents

Assumptions

malstroem makes some assumptions to simplify calculations. The most important ones which you must be aware of:

• malstroem assumes that the terrain is an impermeable surface. This may change in a later version.
• malstroem does not know the concept of time. This means that the capacity of surface streams is infinite no matter the width or depth. Streams wont flow over. The end result is the situation after infinite time, when all water has reached its final destination.
• Water flows from one cell to one other cell (the D8 method).

Example usage

Calculate all derived data for 10mm and 30mm rain incidents ignoring bluespots where the maximum water depth is less than 5cm:

malstroem complete -r 10 -r 30 -filter 'maxdepth > 0.5' -dem dem.tif -outdir c:\outputdirectory

Installation

Theoretically:

pip install cython numpy scipy gdal
pip install git+https://github.com/Kortforsyningen/malstroem.git[speedups]

Unfortunately the above doesn't work on all platforms as malstroem uses som third party libraries and has some optimized code which needs to be compiled for each platform. This method should work on most linux distributions.

Installing on Windows

These instructions are for Python v2.7 64bit. Change accordingly if you prefer another version of Python.

1. Download and install latest Python 2.7 "Windows x86-64 MSI installer"
2. Download and install "Microsoft Visual C++ Compiler for Python 2.7"
3. Go to Christoph Gohlke and download numpy, gdal, cython and scipy wheels matching your python. For Python 2.7 64 bit it should be files ending in cp27‑cp27m‑win_amd64.whl
4. Open windows command prompt and go to the scripts folder in your Python installation. In a defaut install it should be something like

cd c:\Python27\scripts

5. For each of the 4 wheel files downloaded from Gholke (starting with numpy) install it with pip like this:

pip install c:\downloads\numpy‑1.11.3+mkl‑cp27‑cp27m‑win_amd64.whl

6. Clone malstroem from github

git clone https://github.com/Kortforsyningen/malstroem

7. Now (finally) install malstroem

pip install c:\downloads\malstroem[speedups]

8. Still in the scripts folder of your Python (c:\python27\scripts) check that malstroem responds to

malstroem --help

Installing on Mac OSX

The biggest problem on OSX is getting GDAL to work. One known solution is via homebrew

1. Make sure homebrew is installed and you know how to use its Python (See for instance this guide)
2. Install GDAL and its Python bindings

brew install gdal

3. Make sure you use the homebrew Python and install malstroem and its dependencies (If you are using a virtualenv create it using --system-site-packages)

pip install cython numpy scipy
pip install git+https://github.com/Kortforsyningen/malstroem.git[speedups]

Bugs and contributions

• Please report issues using the issue tracker: github.com/Kortforsyningen/malstroem/issues
• Contributions are welcome at github.com/Kortforsyningen/malstroem/

If you are not familiar with GitHub please read this short guide.

License

Copyright (c) 2016
Developed by Septima.dk and Thomas Balstrøm (University of Copenhagen) for the Danish Agency for
Data Supply and Efficiency. This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by the Free Software Foundation,
either version 2 of the License, or (at you option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without
even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PORPOSE. See the GNU Gene-
ral Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not,
see http://www.gnu.org/licenses/.

malstroem's Issues

Automatic build of documentation

Maybe use https://docs.readthedocs.io/en/latest/getting_started.html

Check quotation marks in CLI help

Når man ser hjælpe-syntaksen står filter-options i ' ' , men det skal være i " " for at det virker.

Are there differences between win and nix here?

Wrong program name reported in cli usage text on windows

On windows malstroem --help prints

Usage: malstroem-script.py [OPTIONS] COMMAND [ARGS]...

Wheras the correct usage is

Usage: malstroem [OPTIONS] COMMAND [ARGS]...

This is an issue with clicks autodetection of program name and is discussed here
http://stackoverflow.com/questions/32553969/modify-usage-string-on-click-command-line-interface-on-windows
and here
pallets/click#365

Change width of some vector output fields

C:\Python\Python27\Scripts>malstroem complete -r 30 -r 120 -filter "maxdepth > 0.1" -dem d:\data\GC\Malstroem\dembldg10000.tif -outdir d:\data\GC\malstroem\outputs
Processing
   dem: d:\data\GuyCarpenter\Malstroem\dembldg10000.tif
   outdir: d:\data\guycarpenter\malstroem\outputs
   rain: 30.0mm, 120.0mm
   accum: False
   filter: maxdepth > 0.1
Calculating filled DEM
Calculating bluespot depths
Calculating flow directions
Done
Calculating unfiltered bluespots
Number of bluespots found before filtering: 5182786
Calculating filtered bluespots
Number of bluespots left after filtering: 335159
Calculating watersheds
Calculating pour points at min filled
Writing 335160 pour points
Warning 1: Value 471408033.28000224 of field wshed_area of feature 0 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 1413494643.2000067 of field bspot_area of feature 0 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 104716625.81750049 of field bspot_vol of feature 67501 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 107602733.51000051 of field bspot_vol of feature 146062 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 106728499.5525005 of field bspot_vol of feature 191041 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 111468170.38250053 of field bspot_vol of feature 249098 not successfully written. Possibly due to too larger number with respect to field width
Done
Read input data
Processing stream network
Writing 418968 nodes
Warning 1: Value 471408033.28000224 of field wshed_area of feature 0 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 1413494643.2000067 of field bspot_area of feature 0 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 104716625.81750049 of field bspot_vol of feature 67501 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 107602733.51000051 of field bspot_vol of feature 146062 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 106728499.5525005 of field bspot_vol of feature 191041 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 111468170.38250053 of field bspot_vol of feature 249098 not successfully written. Possibly due to too larger number with respect to field width
Reading input nodes
Creating stream network
Calculating rain events
  30.0mm
  120.0mm
Writing output
Warning 1: Value 471408033.28000224 of field wshed_area of feature 0 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 1413494643.2000067 of field bspot_area of feature 0 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 104716625.81750049 of field bspot_vol of feature 67501 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 107602733.51000051 of field bspot_vol of feature 146062 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 106728499.5525005 of field bspot_vol of feature 191041 not successfully written. Possibly due to too larger number with respect to field width
Warning 1: Value 111468170.38250053 of field bspot_vol of feature 249098 not successfully written. Possibly due to too larger number with respect to field width
Done

Question of course is if these numbers are bigger than what should be expected.

Rename bluespot output from malstroem complete

malstroem complete outputs raster bluespots in a file called labeled.tif. Name this file bluespots.tif instead.

Also rename depths.tif til bs_depths.tif.

Add output datatype to documentation

Add datatypes to output descriptions (rasters and vector attribute tables) in the CLI user guide

@click_log.init()

click_log package that is installed together with malstroem is not supporting following command in current version 0.2.1.

@click_log.init()

Solution is to comment out this line, but than I loose info about the process while running malstroem complete command.

Raster output from malstroem bspots does not have nodata=0

malstroem complete outputs bluespots raster where areas that are not bluespots are 'nodata'. malstroem bspots should do the same.

Assert input data are as expected

• When a method takes more than one raster they need to have the same dimensions, CRS etc. Assert that this is the case
• Input pourpoints and raster bluespots are consistent
• ...

Do these checks and report in CLI to give the user a chance to know what is going wrong.

Set up continuous integration

Using the tox configuration it should be pretty straight forward: http://stackoverflow.com/questions/37016688/how-to-run-tox-with-travis-ci

Getting GDAL python bindings in travis may be complicated. Maybe travis-ci/travis-ci#2938 can help.

Example run

The example run malstroem complete -r 10 -r 30 -filter 'maxdepth > 0.5' -dem dem.tif -outdir c:\outputdirectory returns an ExceptionError due to the presence of the ' character. The model runs normally if the ' characters are removed and the order turns into malstroem complete -r 10 -r 30 -filter maxdepth > 0.5 -dem dem.tif -outdir c:\outputdirectory

Documentation improvements

• Write 'About/History' section?
• Maybe change author from 'Asger Skovbo Petersen' to something else?

Python 2.7 end of life

Are there plans to migrate / shift this program to python 3.x? With the EOL scheduled for 2020, installation might soon be problematic

Refactor vector writing

Lets just assume, that we will be using OGR in one way or the other. This allows us to optimize.