jblindsay / whitebox-tools

An advanced geospatial data analysis platform

*This page is related to the stand-alone command-line program and Python scripting API for geospatial analysis, WhiteboxTools.

The official WhiteboxTools User Manual can be found at this link.

Contents

1. Description
2. Getting Help
3. Downloads and Installation
4. Building From Source Code

1 Description

WhiteboxTools is an advanced geospatial data analysis platform developed by Prof. John Lindsay (webpage; jblindsay) at the University of Guelph's Geomorphometry and Hydrogeomatics Research Group. WhiteboxTools can be used to perform common geographical information systems (GIS) analysis operations, such as cost-distance analysis, distance buffering, and raster reclassification. Remote sensing and image processing tasks include image enhancement (e.g. panchromatic sharpening, contrast adjustments), image mosaicing, numerous filtering operations, classification, and common image transformations. WhiteboxTools also contains advanced tooling for spatial hydrological analysis (e.g. flow-accumulation, watershed delineation, stream network analysis, sink removal), terrain analysis (e.g. common terrain indices such as slope, curvatures, wetness index, hillshading; hypsometric analysis; multi-scale topographic position analysis), and LiDAR data processing. LiDAR point clouds can be interrogated (LidarInfo, LidarHistogram), segmented, tiled and joined, analyized for outliers, interpolated to rasters (DEMs, intensity images), and ground-points can be classified or filtered. WhiteboxTools is not a cartographic or spatial data visualization package; instead it is meant to serve as an analytical backend for other data visualization software, mainly GIS.

2 Getting help

WhiteboxToos possesses extensive help documentation. Users are referred to the User Manual located on www.whiteboxgeo.com.

3 Pre-compiled binaries

WhiteboxTools is a stand-alone executable command-line program with no actual installation. If you intend to use the Python programming interface for WhiteboxTools you will need to have Python 3 (or higher) installed. Pre-compiled binaries can be downloaded from the Whitebox Geospatial Inc. website with support for various operating systems.

4 Building from source code

It is likely that WhiteboxTools will work on a wider variety of operating systems and architectures than the distributed binary files. If you do not find your operating system/architecture in the list of available WhiteboxTool binaries, then compilation from source code will be necessary. WhiteboxTools can be compiled from the source code with the following steps:

1. Install the Rust compiler; Rustup is recommended for this purpose. Further instruction can be found at this link.

2. Download the WhiteboxTools from this GitHub repo.

3. Decompress the zipped download file.

4. Open a terminal (command prompt) window and change the working directory to the `whitebox-tools` folder:

cd /path/to/folder/whitebox-tools/

5. Finally, use the Python build.py script to compile the code:

python build.py

Depending on your system, the compilation may take several minutes. Also depending on your system, it may be necessary to use the `python3` command instead. When completed, the script will have created a new `WBT` folder within `whitebox-tools`. This folder will contain all of the files needed to run the program, including the main Whitebox executable file (whitebox_tools.exe), the Whitebox Runner GUI application, and the various plugins.

Be sure to follow the instructions for installing Rust carefully. In particular, if you are installing on MS Windows, you must have a linker installed prior to installing the Rust compiler (rustc). The Rust webpage recommends either the **MS Visual C++ 2015 Build Tools** or the GNU equivalent and offers details for each installation approach. You should also consider using **RustUp** to install the Rust compiler.

whitebox-tools's Issues

vector_points_to_raster throws error when run without a base raster

Error message:

thread 'main' panicked at 'called Option::unwrap() on a None value', libcore\option.rs:335:21

Edit: might be simply that a base raster is required now, but it used to be optional in Whitebox.

Stream Network tools not running

When running any of the 'Stream Network' tools (e.g. long_profile, length_of_upstream_channels, etc.) I receive the following error:

thread 'main' panicked at 'Colour map not present in Paletted TIFF.', src\main.rs:49:21

Python call back:
.\whitebox_tools.exe --run="Longprofile" --wd="E:\Burnett_2018_Projects\End_of_Fish\Stream_Delin" --d8_pntr='hja_recon_v10_d8recon2_d8pntr.tif' --streams='hja_recon_v10_d8recon2_d8strm.tif' --dem='hja_recon_v10_d8recon2.tif' --output='hja_recon_v10_d8recon2_d8profile.html' -v

Workflow:
input a custom DEM
d8_pntr is output from 'flow_accumulation_full_workflow'
streams is output from 'extract_streams' and based on 'd8_flow_accum' from 'flow_accumulation_full_workflow'

Notes:
d8_pntr and d8_flow_accum rasters appear normal in GIS

System: Windows 10 x64
Ram: 64 GB
Python version: 3.6
WBT Version: 0.7

Problem writing denoised tif

Hello

I tried to denoise a DEM using the following command

C:/WhiteboxTools/whitebox_tools.exe -r=FeaturePreservingDenoise -v --dem="/path/dem.tif" -o=demf.tif --filter=15 --norm_diff=20.0 --num_iter=4

the product is a DEM with a similar weight as the original one, however, I have been unable to read it with GDAL. Running gdalinfo on the output produces the following:

gdalinfo demf.tif
ERROR 1: demf.tif:Not a TIFF file, bad version number 10752 (0x2a00)
gdalinfo failed - unable to open 'demf.tif'.

The output of gdalinfo for the original file is the following

gdalinfo "/path/dem.tif"
Driver: GTiff/GeoTIFF
Files: /path/dem.tif
Size is 9001, 9001
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (21.999944444444445,45.000055555555555)
Pixel Size = (0.000111111111111,-0.000111111111111)
Metadata:
  AREA_OR_POINT=Point
  TIFFTAG_DATETIME=2015:05:26 21:46:30
  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
  TIFFTAG_SOFTWARE=[sensitive]
  TIFFTAG_XRESOLUTION=50
  TIFFTAG_YRESOLUTION=50
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  21.9999444,  45.0000556)
Lower Left  (  21.9999444,  43.9999444)
Upper Right (  23.0000556,  45.0000556)
Lower Right (  23.0000556,  43.9999444)
Center      (  22.5000000,  44.5000000)
Band 1 Block=9001x1 Type=Float32, ColorInterp=Gray

I hope this info is helpful, I tried to report this problem as instructed in the documentation

Kind regards,
Ignacio

Add Numpy Array support

It would be a great feature if WhiteboxTools could support Numpy Arrays both as input and output. That would simplify integration with other Python librairies and offer a way to keep the data in memory for further processing.

VectorPolygonsToRaster issue - WhiteboxTools v1.0.0

The WhiteboxTools "VectorPolygonsToRaster" still returns an output .tis with gaps.
The * Release Notes: * specify:

• Fixed a bug in the VectorPolygonsToRaster tool that sometimes mishandled polygon holes.

Does not to be fixed. See attached sample

win10 error: Reading data... 'gbk' codec can't decode byte 0x82 in position 59: illegal multibyte sequence 1

elevation_above_stream_euclidean producing empty tif

I have a script that uses both the elevation_above_stream_euclidean and the elevation_above_stream tools. The inputs are the same.

These are the lines of code:

wbt.elevation_above_stream(dem= path+name+tif, streams= name + '_'+ n + 'streams.tif', output= name + '' + n + 'HAND.tif')

wbt.elevation_above_stream_euclidean(dem= path+name+tif, streams=name + '_' + n + 'streams.tif', output= name + '' + n + 'HANDe.tif')

However, the wbt.elevation_above_stream_euclidean produces an empty .tif without throwing an error.

build errors

I was attempting to build the latest version of whitebox_tools (commit df41d62) using the latest version of rust. I received the following errors. I'm not familiar with Rust. Is some other build option needed?

Thanks

--

rustup 1.19.0 (2af131cf9 2019-09-08)
rustc 1.37.0 (eae3437df 2019-08-13)

> cargo build --release

error: attempt to shift left with overflow
   --> src/tools/lidar_analysis/lidar_idw_interpolation.rs:686:79
    |
686 |                                                         frs.insert(p.x, p.y, ((255 << 24) | (clr.blue << 16) | (clr.green << 8) | clr.red) as f64);
    |                                                                               ^^^^^^^^^^^
    |
    = note: #[deny(exceeding_bitshifts)] on by default

error: attempt to shift left with overflow
   --> src/tools/lidar_analysis/lidar_idw_interpolation.rs:686:93
    |
686 |                                                         frs.insert(p.x, p.y, ((255 << 24) | (clr.blue << 16) | (clr.green << 8) | clr.red) as f64);
    |                                                                                             ^^^^^^^^^^^^^^^^

error: attempt to shift left with overflow
   --> src/tools/lidar_analysis/lidar_idw_interpolation.rs:806:52
    |
806 |                                             val = ((255 << 24) | ((val_blue.round() as u16) << 16) | ((val_green.round() as u16) << 8) | (val_red.round() a
s u16)) as f64;
    |                                                    ^^^^^^^^^^^

error: attempt to shift left with overflow
   --> src/tools/lidar_analysis/lidar_idw_interpolation.rs:806:66
    |
806 |                                             val = ((255 << 24) | ((val_blue.round() as u16) << 16) | ((val_green.round() as u16) << 8) | (val_red.round() a
s u16)) as f64;
    |                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: attempt to shift left with overflow
   --> src/tools/lidar_analysis/lidar_idw_interpolation.rs:871:60
    |
871 |                                                     val = ((255 << 24) | ((val_blue.round() as u16) << 16) | ((val_green.round() as u16) << 8) | (val_red.r
ound() as u16)) as f64;
    |                                                            ^^^^^^^^^^^

error: attempt to shift left with overflow
   --> src/tools/lidar_analysis/lidar_idw_interpolation.rs:871:74
    |
871 |                                                     val = ((255 << 24) | ((val_blue.round() as u16) << 16) | ((val_green.round() as u16) << 8) | (val_red.r
ound() as u16)) as f64;
    |                                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: attempt to shift left with overflow
   --> src/tools/lidar_analysis/lidar_nn_gridding.rs:665:79
    |
665 |                                                         frs.insert(p.x, p.y, ((255 << 24) | (clr.blue << 16) | (clr.green << 8) | clr.red) as f64);

Unrecognized tool name FeaturePreservingDenoise

Hello!
There is one error when I use "FeaturePreservingDenoise".

./whitebox_tools -r=FeaturePreservingDenoise -v --wd=/home/keo/nnd_storage/thirdpart_software/whitebox-tools/testdata/test_dsm/ -i=DEM_gaps_filled.tif -o=DEM_smoothed.tif --filter=15 --norm_diff=20.0 --num_iter=4
thread 'main' panicked at 'Unrecognized tool name FeaturePreservingDenoise.', src/main.rs:66:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

Docker image

It would be useful to have a docker image of this, so you could pull-down and run a version of whitebox-tools in environments that lack rust (typed from a Windowz desktop that has docker but no admin rights let alone a rust compiler). Happy to help with this if I get the 👍

RasterArea tool

The new RasterArea Tool is missing in the whitebox_tools.py.

All pixels shift to the right in very large rasters

When using either BreachDepressions or D8FlowAccumulation on a large raster exceeding 500 000 000 pixels, the resulting raster will be shifted one pixel to the right.

Breach Depressions (not fast)

Is there an updated function to do the breach depressions (the not "fast") version?

Inconsistent option names in the Extract Valleys tool

In the "Extract Valleys" tool there is a --variant parameter which accoding to the description accepts one of the following values: LQ (lower quartile), JandR (Johnston and Rosenfeld) or PandD (Peucker and Douglas), and uses LQ as default. However in the JSON tool description available values listed in the short form, while defualt value listed in the long form Lower Quartile.

{"name":"Variant","flags":["--variant"],"description":"Options include 'LQ' (lower quartile), 'JandR' (Johnston and Rosenfeld), and 'PandD' (Peucker and Douglas); default is 'LQ'.","parameter_type":{"OptionList":["LQ","JandR","PandD"]},"default_value":"Lower Quartile","optional":false}

Not sure is this bug or default value was just missed during values change.

Can't produce TIF from FLT/SDAT with Fill Single-Cell Pits or Breach Depressions

The Fill Single-Cell Pits and Breach Depressions tools seem to be unable to produce a GeoTIFF if the input is FLT or SDAT.

wbt = WhiteboxTools()
wbt.fill_single_cell_pits("input.tif", "fromTIF.flt")
# ...
# Saving data...
# error while writing: Custom { kind: InvalidData, error: StringError("Error while writing GeoTIFF file.") }
# Output file written
# ...

Are those tools trying to write spatial reference data to the TIFF file, but crash because they can't find any with FLT or SDAT files? I tried the same thing with Breach Single-Cell Pits and the TIFF file is correctly generated (without any spatial reference as expected).

Provide a set_callback method which sets a new global callback

I want to override the default callback for all functions so it would be useful to have a set_callback method in order to do this once rather than when each function is called.

VectorPolygonsToRaster issue

• whitebox version: 0.9.0
• Python version: 3.7.3
• Operating System: Windows 10

Description

The whitebox tool "VectorPolygonsToRaster" return a different result compare to the Whitebox GAT -win tool "Vector polygons to raster" . The output .tif has gaps!!!

What I Did

from whitebox import WhiteboxTools

# Set up the environment
wbt = WhiteboxTools()
data_dir = "/../whitebox_issue"
wbt.work_dir = (data_dir) 

i_vector = "poly.shp"
fieldZ = "Value"
out_raster = "poly2raster_PYTHON.tif"

wbt.vector_polygons_to_raster(
    i=i_vector,
    output=out_raster,
    field=fieldZ,
    nodata=True,
    cell_size=1,
   # base=None,
   # callback=default_callback
)

unstable library feature 'iterator_step_by' error when compiling

I wanted to update whitebox-tools but I failed to compile:

Compiling whitebox_tools v0.12.0 (file:///../Models/whitebox-tools)
error[E0658]: use of unstable library feature 'iterator_step_by': unstable replacement of Range::step_by (see issue #27741)
  --> src/tools/gis_analysis/construct_vector_tin.rs:337:46
   |
337 |         for i in (0..result.triangles.len()).step_by(3) {
   |                                              ^^^^^^^

error[E0658]: use of unstable library feature 'iterator_step_by': unstable replacement of Range::step_by (see issue #27741)
  --> src/tools/lidar_analysis/lidar_construct_vector_tin.rs:434:58
   |
434 |                     for i in (0..result.triangles.len()).step_by(3) {
   |                                                          ^^^^^^^

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0658`.
error: Could not compile `whitebox_tools`.

After running rustc --explain E0658, it seems that I may use a nightly version of rustc to be able to use the iterator_step_by.

VectorAttributeField parameter improvements

There is new VectorAttributeField parameter in WhiteboxTools which represents attribute field of the vector layer. May I suggest couple of improvements for it:

1. add information about required field data type, e.g. numeric, text, etc. This can be useful when algorithms dealing with some specific data types will be added.
2. add information about parent layer. This will be useful if algorithm accepts multiple vectors and fields.

Whiteboxtools and QGIS error while writing:

Hello, whiteboxtools v1.02, it appears the tools run but wont write to a file whether I save it to temporary file or permanent file. Note I've reproduced the following error below with multiple tools. This is running on Windows through QGIS.

Reading data...

Saving data...

error while writing: Custom { kind: InvalidData, error: "Error while writing GeoTIFF file." }

Output file written

Elapsed Time (excluding I/O): 3.8

MosaicWithFeathering discarding GeoTIFF coordinate system

I am trying to create mosaics of GeoTIFF files. I am obtaining a resulting GeoTIFF that looks ok, but the coordinate system is then gone in the resulting mosaic.

How can I georefence properly the resulting mosaic?

Inputs:

Driver: GTiff/GeoTIFF
Files: /tmp/out22_g_nd.tif
Size is 1024, 1024
Coordinate System is:
PROJCS["WGS 84 / UTM zone 29N",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",-9],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","32629"]]
Origin = (579085.567641534260474,4738466.775685614906251)
Pixel Size = (1.502461069483093,-1.502458127183625)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
  TIFFTAG_SOFTWARE=WhiteboxTools
  TIFFTAG_XRESOLUTION=72
  TIFFTAG_YRESOLUTION=72
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  579085.568, 4738466.776) (  8d 1'58.59"W, 42d47'40.80"N)
Lower Left  (  579085.568, 4736928.259) (  8d 1'59.37"W, 42d46'50.93"N)
Upper Right (  580624.088, 4738466.776) (  8d 0'50.87"W, 42d47'40.22"N)
Lower Right (  580624.088, 4736928.259) (  8d 0'51.67"W, 42d46'50.35"N)
Center      (  579854.828, 4737697.517) (  8d 1'25.12"W, 42d47'15.58"N)
Band 1 Block=1024x1 Type=Float32, ColorInterp=Gray
    Computed Min/Max=11.000,251.000
  NoData Value=0

Mosaic:

Driver: GTiff/GeoTIFF
Files: mosaic.tiff
Size is 1282, 1296
Coordinate System is:
LOCAL_CS["unnamed",
    UNIT["unknown",1]]
Origin = (577458.933401877293363,4738466.775685614906251)
Pixel Size = (2.469379546168251,-2.469373084423899)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
  TIFFTAG_SOFTWARE=WhiteboxTools
  TIFFTAG_XRESOLUTION=72
  TIFFTAG_YRESOLUTION=72
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (  577458.933, 4738466.776) 
Lower Left  (  577458.933, 4735266.468) 
Upper Right (  580624.678, 4738466.776) 
Lower Right (  580624.678, 4735266.468) 
Center      (  579041.806, 4736866.622) 
Band 1 Block=1282x1 Type=Byte, ColorInterp=Red
    Computed Min/Max=9.000,255.000
  NoData Value=0
Band 2 Block=1282x1 Type=Byte, ColorInterp=Green
    Computed Min/Max=6.000,255.000
  NoData Value=0
Band 3 Block=1282x1 Type=Byte, ColorInterp=Blue
    Computed Min/Max=2.000,255.000
  NoData Value=0
Band 4 Block=1282x1 Type=Byte, ColorInterp=Alpha
    Computed Min/Max=255.000,255.000
  NoData Value=0

LidarJoin scrambling points.

I ran a test of LidarJoin using WhiteboxTools Runner on 5 files of total point record size 19753893. Version 1.2. These were non RGB points. The data produced in the output file was scrambled and grouped at the XY, ZX and ZY - see attached png file. I tried just 2 smaller files and returned the same result. I used lastools lasmerge and received the correct merge.

raster projections - low priority

Hello,

First of all thank you very much for the Python API to your tools. We do tree species identification so we need to generate canopy height models using lidar DTM and DSM (I would love to be able to subtract the results of the two lidar_tin_gridding operations without saving the intermediate rasters but I digress!)

Is there a way to set the projection on the output rasters to the one from the input las files? I searched in the manual but could not find any information.

Development Plan with GAT

GREAT WORK JOHN! woot.

Just curious if you plan to consolidate any common functions of whitebox-GAT to whitebox tools? Is this already the case that Whitebox-GAT calls rust whitebox-tools? I'm excited about this project -
myself I'd like to eventually be able to call rust these rust functions directly for go-code.. can just use go as a scripting language and call directly into the Whitebox-Tools binary for the time being... Do function calls return json output for non-file output? I imagine that there are a number of functions which have small amount of output, it'd be nice if this was machine parseable.

go john go!

Cost distance produces integer data type

Encountered a strange problem while trying to run Cost Distance tool. The output is consistently an integer data type instead of a float point. Currently working on v0.15.0 and here are the source and cost surfaces. I can't seem to find any issue with the raw data. Any ideas?

Watershed w/ raster pour_pts as input

Hoping to have Watershed enhanced so that rasters can be used as pour_pts similar to Watershed in Whitebox. This is useful for generating “lakesheds” via scripts.

FlowAccumulationFullWorkflow not ignoring 'No Data'

When running the 'FlowAccumulationFullWorkflow' with a geotiff DEM input, the flow pointer and flow accumulation processes produce data in the DEM 'NoData' areas. The 'NoData' areas are recognized in both ArcGIS and GDAL. This particular issue does not appear in WhiteboxGAT when using the same input DEM.

I did not test on the independent d8 flow pointer and d8 flow accumulation functions.

OS: Windows 10 Pro
Python Version: 3.60
Whitebox Tools Version: 0.70

Resample tool needs existing file as output

Seems output parameter in the Resampling tool defined incorrectly, it expects existing file to be passed as output, not new file

$ whitebox_tools --toolparameters=Resample
{"parameters":[{"name":"Input Files","flags":["-i","--inputs"],"description":"Input raster files.","parameter_type":{"FileList":"Raster"},"default_value":null,"optional":false},{"name":"Destination File","flags":["--destination"],"description":"Destination raster file.","parameter_type":{"ExistingFile":"Raster"},"default_value":null,"optional":false},{"name":"Resampling Method","flags":["--method"],"description":"Resampling method","parameter_type":{"OptionList":["nn","bilinear","cc"]},"default_value":"cc","optional":true}]}

Is this intentional or maybe a typo?

SDAT file flipped horizontally at export

When converting a DEP file to SDAT format, the exported raster is flipped horizontally. The problem is only with the Python API as the SDAT export form Whitebox GAT 3.4 doesn't have this problem.

wbt.convert_raster_format('infile.dep', 'outfile.sdat')

I tested with the lastest compiled binaries (0.16.0). Input and output exemple files can be downloaded here.

0.15 BreachDepressions produces unmodfied output

When i run breach depressions I get output that is identical to the input. I tested on Windows and Linux. 0.14 works correctly.

Interpolation extent

Hello,

I am working with another Python package (pyfor) that exhibits similar behaviour so am recycling a large part of my post from there, I hope that is OK

I am trying to understand the behaviour of the interpolation when I am creating surfaces such as a DTM. As mentionned in my email discussion, our point clouds are already classified so I filter out the ground returns then use the following in pyyhon to create the DTM raster.

wbt.lidar_tin_gridding( i = name, output = normpath(DTM_dir + "/" + las_basefilename + "_DTM.tif"), parameter="elevation", returns="all", resolution=0.25, exclude_cls='1,3,4,5,7,8,9,18')

We are processing 1600 buffered 1km^2 las tiles (40m buffer) to create DTM, DSM and CHM which then gets used by our ITC segmentation script. The rasters are at 25cm resolution so the output raster tiles should be 4160*4160. Our previous method using the lidR package in R produced this result. We are super happy with the speed of wbt (20 seconds vs 20 minutes in R...), it is exactly what we need. I think my supervisor (Benoît St-Onge at UQAM) and you should have a chat one day by the way!

When I use whitebox (and pyfor), the rasters have uneven dimensions such as 4079*4159 for example. I understand what is happening, the ground las points do not reach the corner of the declared extent of the las file and the interpolation stops a bit short. Here are some screenshots that explain the issue

LAS ground points (1000m * 1000m)

lidR output (4160*4160)

pyfor output (4079*4079)

whitebox output for comparison (4080*4080)

wbt seems to miss the top pixels near the extent edge

Arcmap TIN display

I believe what may be happening is the interpolation (pyfor uses scipy, do not know what wbt uses) is not getting the extent from the LAS file but calculating it from the actual points which makes it fall short of the 'official' extent. Do not know what R does different in this respect but this would be the desired output.

Any way I can force this to be the case?

Thank you!

question about crates - geo

Hello, John

I have been spending some time looking through the code and wanted to ask a seemingly simple question.
After reading though the FAQ and some of the closed issues raised about GDAL, it sounds like you are investing in writing a lot of the functionality from the ground up solely in Rust.
I was wondering if there was any use or plans for any of the crates in the geoRust group?. I didn't see it referenced in the Cargo.toml and was wondering what your thoughts were on those projects?

Thank you for your work, I hope to contribute once I get more comfortable with the language.

Pixel misalignment between original and resulting images

Using the testdata/DEM.tif, I noticed that the resulting images from most tools shifted half a pixel towards the lower-right corner. The issue can be reproduced using the script below.

wbt = WhiteboxTools()
wbt.feature_preserving_denoise("DEM.tif", "smoothed.tif", filter=9)
wbt.breach_depressions("smoothed.tif", "breached.tif")
wbt.d_inf_flow_accumulation("breached.tif", "flow_accum.tif")

For example:
DEM.tif
CRS | EPSG:4326 - WGS 84 - Geographic
Extent | 664692.0589773271931335,4878903.5508630042895675 : 686018.1474491551052779,4895827.5998771572485566
Pixel Size | 89.9835,-90.0215

smoothed.tif
CRS | EPSG:4326 - WGS 84 - Geographic
Extent | 664737.0507251580711454,4878858.5400943495333195 : 686063.1391969859832898,4895782.5891085024923086
Pixel Size | 89.9835,-90.0215

diff = ~ 45 (half a pixel)

LidarTINGridding kernel panic

Hello, I am getting these errors when running the following code, which works on other lidar datasets I have. The process continues for a few more tiles with this message interspersed with the progress messages and then the program hangs with no output.

Any idea where I can start troubleshooting these lidar tiles? Thank you. I tried resolution of 5m but same issues.

wbt.lidar_tin_gridding( parameter="elevation", returns="all", resolution=0.25, exclude_cls='0,1,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18')

Progress: 5%
Finished interpolating \\374000_5141000_suffix (9 of 138)
Finished interpolating \\375000_5143000_suffix (10 of 138)
Progress: 6%
thread 'unnamed' panicked at 'index out of bounds: the len is 604504064 but the index is 604504064',/rustc/9fda7c2237db910e41d6a712e9a2139b352e558b\src\libcore\slice\mod.rs:2463:10
note: Run with RUST_BACKTRACE=1 for a backtrace.
thread 'unnamed' panicked at 'index out of bounds: the len is 604504064 but the index is 604504064',/rustc/9fda7c2237db910e41d6a712e9a2139b352e558b\src\libcore\slice\mod.rs:2463:10
Finished interpolating \\375000_5139000_suffix (11 of 138)
Progress: 7%
thread unnamed' panicked at 'index out of bounds: the len is 604504064 but the index is 604504064',/rustc/9fda7c2237db910e41d6a712e9a2139b352e558b\src\libcore\slice\mod.rs:2463:10
thread 'unnamed' panicked at 'index out of bounds: the len is 604504064 but the index is 604504064',/rustc/9fda7c2237db910e41d6a712e9a2139b352e558b\src\libcore\slice\mod.rs:2463:10

Adding more options for LiDAR Interpolation

Are there any plans to allow the LiDAR interpolation functions to use any arbitrary field rather than what is currently available ('elevation', 'intensity', 'class', 'scan angle', 'user data'). The interpolation functions work great, but it would be helpful to be able to apply them to e.g., the number of returns per pulse (to get average number of returns per pulse in a cell), or to the RBG fields (in order to extract a color image) without having to write more code to bin these values. Right now the only way to do this is to trick the function by changing e.g., swapping the headers for Z and number of returns.

Problem running NewRasterFromBase on Windows

Having identified several issues with the input parameters that can cause problems on Windows (#20), and also realised that --input is not actually the right parameter, I'm still unable to get NewRasterFromBase to run on Windows:

C:\Processing>whitebox_tools.exe --version
whitebox-tools v0.11.0 by Dr. John B. Lindsay (c) 2017-2018

C:\Processing>whitebox_tools.exe --run=NewRasterFromBase --wd="C:\Processing\" -i="simple.tif" --output="simple57.tif" --value=nodata --data_type=integer
thread 'main' panicked at 'Tool run with no paramters.', src\main.rs:76:21
stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: BaseThreadInitThunk
  15: RtlUserThreadStart

I do not believe the input file is the problem - it works with the Linux whitebox_tools, and I have tried a variety of files, all failing.

Is there anything else obvious I'm missing? Maybe it is just an issue with the order in which I've specified the parameters - there is probably some combination I haven't tried.
Would a proper debug build for Windows help?

another "panick" issue for `option::unwrap()`

trying to fill depressions in a DEM file. my code is below. the same thing happens when I call the 'slope' function. any thoughts on how to fix this?

wbts = wbt()
wbts.fill_depressions(dem='/Raster/DEM8m/apa8mdem',
output= 'apDEMfill.tif',
fix_flats=True)

returns:
./whitebox_tools --run="FillDepressions" --dem='/Raster/DEM8m/apa8mdem' --output='apDEMfill.tif' --fix_flats -v

---

• Welcome to FillDepressions *

---

Reading data...
thread 'main' panicked at 'called Option::unwrap() on a None value', src/libcore/option.rs:345:21
note: Run with RUST_BACKTRACE=1 environment variable to display a backtrace.

output: 0

Pixel shift when going from FLT to TIF (multiple tools)

Several tools shift by half-pixel the output raster:

• fill_single_cell_pits => half-pixel shift (down-left)
• breach_single_cell_pits => half-pixel shift (down-left)
• breach_depressions => half-pixel shift (down-left)
• convert_raster_format => no shift from FLT or TIFF, but half-pixel shift (down-left) if going from DEP to FLT or TIFF

The problem occurs when going from FLT to FLT or TIFF. If the input raster is TIFF, the output raster (TIFF or FLT) has the correct extent.

Projection Issue - Low Priority

DEM Geotiffs with NAD 83 UTM Zone 10N projection produce derivative geotiffs with grids in the same coordinate system, but the metadata has changed to GCS NAD 1983 with decimal degree units, which causes difficulties in GIS.

Filename changed at export when containing the extension itself

When using tool convert_raster_format, if the name of the exported file contains it's extension, the companion file won't be named correclty.

wbt.convert_raster_format('infile.dep', '0.sdat_outfile.sdat')
# will give the files:
#   0.sdat_outfile.sdat
#   0.sgrd_outfile.sgrd instead of 0.sdat_outfile.sgrd

The problem is there also with .flt (companion file .hdr) and .rst (companion file .rdc). The problem is only with the Python API as the files exported from Whitebox GAT 3.4 doesn't experience this problem.

watershed only takes shapefile as pour point input

Which seems a bit inconsistent when necessary pre processing tools like snap pour points don't output shapefiles and raster to vector points is not yet ported.

Permission denied in QGIS Plugin

Hi,

I set up the workflow as described but I cannot use the tool of Whitebox in QGIS within Processing.

These the steps I followed:

• download the tar and extracted in /home/matteo/Download
• added Alex url in QGIS to get the Processing plugin
• set up the plugin in this way:

where WBT is the folder containing the files of the tar

Every algorithm gives me the following error:

/bin/sh: 1: /home/matteo/Downloads/WBT/: Permission denied

I also tried to change the permission of the folder to 777 (but I did all the steps as normal user) and the error is still here.

OS: Linux Debian Sid, QGIS 3.9 cc @alexbruy

Shapefile geometry type support

Not sure, if this is an issue or user error, but I am trying to use WBT clip tool to extract streams (polyline) inside a watershed (polygon) and I get the following error:

Reading data...
thread 'main' panicked at 'Unrecognized ShapeType', src\vector\shapefile\geometry.rs:483:18
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

I checked the geometry types of my datasets: watershed = 'Polygon' and streams = 'LineString', however, after looking through geometry.rs I see no reference to LineString. Is there something I'm missing here?

Additional info. the watershed originally a DEM that is polygonized using rasterio & shapely, and finally written out using geopandas as ESRI Shapefile. And the streams are NHD projected streams. Here is the snippet:

with rasterio.open("D:/dem.tif") as ds_dem:
    arr_dem = ds_dem.read(1)

arr_dem[arr_dem > 0] = 100
mask = arr_dem == 100

results = (
    {'properties': {'raster_val': v}, 'geometry': s}
    for i, (s, v)
    in enumerate(
    shapes(arr_dem, mask=mask, transform=ds_dem.transform)))

poly = list(results)
poly_df = gpd.GeoDataFrame.from_features(poly)
poly_df.crs = in_crs
poly_df.to_file("D:/watershed_boundary.shp")

streams     = "D:/NHD_streams.shp"
boundary    = "D:/watershed_boundary.shp"
output      = "D:/clipped_streams.shp"
wbt.clip(streams, tmp, output)

SplitColourComposite error in v.0.16.0 Windows binaries

Hi @jblindsay

I was testing SplitColourComposite in WhiteboxTools 0.16.0, following this fix

https://github.com/alexbruy/processing-whitebox/issues/17#issuecomment-482542035

but now I'm getting the error below:

C:\>"C:\OSGeo4W64\WhiteboxTools_win_amd64\WBT\whitebox_tools.exe" --run=SplitColourComposite --input="D:\orto_teste.tif" --red="D:\orto_teste_r.tif" --green="D:\orto_teste_g.tif" --blue="D:\orto_teste_b.tif" -v
***********************************
* Welcome to SplitColourComposite *
***********************************
Reading data...
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src\libcore\option.rs:345:21
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

Can this be a local problem?

convert_raster_format adding NoData strips on right if TIFF input

The tool Convert Raster Format is adding a strip of NoData and slightly modified value on the right of the ouput raster when the input format is TIFF (NoData in purple in the image below).

The width of the strip varies: I got 55 pixels on a 10039 x 15151 raster and 63 pixels on a 2623 x 2651 raster (both TIFF) but has always the same look (one on two rows with repetitive diagonals). The values between two rows of NoData is not the original value, sometimes a bit higher sometimes a bit lower. If I convert the original TIFF to FLT with ArcMap and then convert those FLT to FLT or SDAT with Whitebox Tools, no NoData strip is added.

trouble running Viewshed - thread 'main' panicked at 'index 43 out of range for slice of length 34'

thread 'main' panicked at 'index 43 out of range for slice of length 34', libcore/slice/mod.rs:1971:5
note: Run with RUST_BACKTRACE=1 for a backtrace.

here is the tiff.
https://www.dropbox.com/s/wyt545n5mpqpy1m/governer_island_ns2.tif?dl=0

Tried to fill None with zero - did not help the issue,

Inconsistencies in command parsing on Windows

Some issues I've noticed trying to run whitebox_tools on Windows and comparing with Linux:

In general it seems that either --wd needs to be used, or paths need to be provided for input and output files (i.e. whitebox_tools never just looks in the current working directory), but

• On Windows NewRasterFromBase seems to require the use of --wd. At least some other tools such as AbsoluteValue will run without --wd, as long as you specify paths for the input and output files. On Linux NewRasterFromBase is the same as the other tools.
• On Windows --wd needs to be specified before the input and output files. On Linux it seems the order of the parameters does not matter.
• When using --wd on Windows the path needs to include a trailing \ character. On Linux the path can end with the name of the directory.

FillBurn enters infinite loop

When trying to run the FillBurn tool it enters an infinite loop, and soon crashes from running out of RAM (6MB input raster ends up trying to use over 16GB RAM).

This is the loop that goes out of control:
https://github.com/jblindsay/whitebox-tools/blob/master/src/tools/hydro_analysis/fill_burn.rs#L503

It ends up reporting over 30000% progress.

Does this happen for anyone else?

Build with GDAL?

It looks like there are Rust bindings for GDAL. Is it possibly the offload the Raster/Vector IO to that library for more supported file options?

Inconsistence in the help URLs

I noticed small inconsistence in the manual URL's. Looks like for the wast majority of tools group name in the URL constructed by replacing spaces with underscores and converting text to the lower case, e.g.LiDAR Tools -> lidar_tools. But for Math and Stats Tools this is not the case, this group converted to mathand_stats_tools. Is this intentional or there is a minor bug in the URL construction?

Asking because in the WhiteboxTools QGIS plugin I create help links automaticaly using tool name and group and this inconsistence does not allow to generate valid urls for tools from the Math as Stats group. May I ask for some information how help URLs are generated, are my assumptions correct?