lutraconsulting / mdal

Mesh Data Abstraction Library

MDAL

Mesh Data Abstraction Library

MDAL is an OSGeo Community Project

• see Unstructured Mesh Layers
• see 3D layered meshes
• see 1D meshes
• see Mesh frame editing

Mailing List

To discuss development and usage, please use [email protected]

Supported Formats

You can use MDAL to load the following file formats:

• 2DM: Mesh representation of various various hydrodynamic modelling packages (e.g. BASEMENT, TUFLOW)
• XMS TIN: Mesh TIN representation in ASCII format
• NetCDF: Generic format for scientific data. Examples can be found here
• GRIB: Format commonly used in meteorology. Examples can be found here
• XMDF*: Outputs of various hydrodynamic modelling packages (e.g. TUFLOW, HYDRO_AS-2D)
• XDMF*: As an example, hydraulic outputs from BASEMENT 3.x modelling package
• DAT: Outputs of various hydrodynamic modelling packages (e.g. BASEMENT, HYDRO_AS-2D, TUFLOW)
• 3Di*: 3Di modelling package format based on CF Conventions
• UGRID*: Unstructured Grid format based on CF Conventions
• FLO-2D: Outputs of the FLO-2D modelling package
• Selafin/Serafin: Outputs of the TELEMAC 2D hydrodynamic modelling package
• HEC-RAS: Outputs of the HEC-RAS modelling package
• SWW: Outputs of the ANUGA modelling package
• Esri TIN: Format for storing elevation data as a triangulated irregular network
• SAGA FLOW**: Rasters in the SAGA flow direction format
• ADCIRC***: ADCIRC hydrodynamic model results
• PLY: Stanford Polygon Format also useful for mesh created from point clouds by PDAL
• DFSU*:DHI Flexible File Format (External Driver)
• DFS2*:DHI Grid Series File (External Driver)
• H2i*: Outputs of the H2i modelling package
• MIKE 21: Mesh Text/plain file with *.mesh extension

* Data lazy loaded

** Formats can be preprocessed using QGIS Crayfish Mesh processing algorithm to one of supported formats

*** Results should pre pre-processed to become UGRID compliant

Format	Mesh Frame Support	Mesh Lazy Loaded	1D Data Support	2D Data Support	3D Data Support	Data Lazy Loaded
2DM	READ-WRITE	N/A	N/A	N/A	N/A	N/A
XMS TIN	READ-ONLY	N/A	N/A	N/A	N/A	N/A
Esri TIN	READ-ONLY	N/A	N/A	N/A	N/A	N/A
GDAL-NetCDF	READ-ONLY	NO	NO	READ-ONLY	N/A	NO
GDAL-GRIB	READ-ONLY	NO	NO	READ-ONLY	N/A	NO
XMDF	N/A	N/A	NO	READ-ONLY	READ-ONLY	YES
XDMF	N/A	N/A	NO	READ-ONLY	NO	YES
DAT	N/A	N/A	READ-ONLY	READ-WRITE	N/A	NO
3Di	READ-ONLY	NO	READ-ONLY	READ-ONLY	NO	NO
UGRID	READ-WRITE	NO	READ-ONLY	READ-WRITE	NO	NO
FLO-2D	READ-ONLY	NO	READ-ONLY	READ-WRITE	NO	NO
Selafin	READ-WRITE	NO	NO	READ-WRITE	NO	NO
SWW	READ-ONLY	NO	NO	READ-ONLY	NO	NO
PLY	READ-WRITE	N/A	READ-WRITE	READ-WRITE	READ-WRITE	N/A
DFSU	READ-ONLY	N/A	NO	READ-ONLY	READ-ONLY	YES
DFS2	READ-ONLY	N/A	NO	READ-ONLY	N/A	YES
H2i	READ-ONLY	NO	NO	READ-ONLY	N/A	YES
Mike21	READ-WRITE	N/A	N/A	N/A	N/A	N/A

Standalone Installation

MDAL can be installed as a stand-alone package (i.e. outside of QGIS) using conda.

The package can installed by running :

conda install -c conda-forge mdal

This package provides the MDAL ABI through the mdal shared object( i.e. mdal.dll, libmdal.dylib or libmdal.so) and the mdalinfo CLI.

A friendly note about versions. The conda package is usually targetted at the latest version of GDAL on conda-forge. This is usually a later version than used by QGIS. Therefore, there may be some subtle differences in behaviour when loading e.g. GRIB files.

Versioning and integration in QGIS

QGIS contains internal copy of MDAL library in following versions:

QGIS	MDAL	Features
3.0.2	N/A	2D meshes
3.2.3	0.0.3
3.4.14	0.0.10
3.6.3	0.3.2	Many new formats supported
3.8.3	0.3.3
3.10.0	0.3.3
3.10.1	0.4.0	Save datasets for some formats
3.10.2	0.4.1
3.10.3	0.4.2
3.12.0	0.5.1	3D layered meshes
3.14.0	0.6.1	1D meshes
3.16.0	0.7.1	Mesh creation, PLY format
3.18.0	0.8.0	Dynamic drivers, DFSU format
3.20.0	0.8.1
3.22.0	0.9.x	DFS2, DFSU 3D, PLY write
3.24.0	1.0.x	TBD
3.24.1	0.9.90	H2i format
3.28.0	1.0.0	write dataset capability for UGRID format
3.28.2	1.0.1
3.28.4	1.0.2
3.30.0	1.0.2
3.36.0	1.1.0	Mike21 format support read/write
3.38.0	1.2.0	Groundwater / surface water meshes for 3Di format

versions X.Y.9Z are development versions or alpha/beta releases (e.g. 0.4.90, 0.4.91, ...)

Backporting

We maintain the version of MDAL used in current QGIS LTR version. For any CRITICAL bugfixes (e.g. crashes, coredumps, regressions, data corruption) we want to backport the code to MDAL LTR version. To do this, label your pull request with label "backport release-XXX". Once pull request is merged, mdal-bot will automatically create a pull request to specified branch. Note that the pull request must be squashed to 1 commit for automatic backport.

Development

Coding standards & Contribution

MDAL is an open-source project and all contributions to either documentation, format support, testing or code are more than appreciated. Any change to the code must go through Pull Request, followed by review by one of the MDAL core developers.

To be able to accept a pull request, please verify that:

• code follows QGIS coding style
• code is properly unit-tested with a set of small test files under tests/data/<format> (code coverage > 90%)
• code is reasonably documented and easy to read
• code compiles without any compilation warnings
• no dead-code (e.g. unused functions) or commented out code
• all new code or new dependencies (e.g. libraries) have GPLv2 compatible license
• all tests pass
• the code if properly formated (run ./scripts/check_all.bash)

Please respect our Contributor Covenant Code of Conduct

Build

Windows

For 64-bit version:

• Install Microsoft Visual Studio 16 2019
• Install OSGeo4W (64bit) to C:\OSGeo4W64
• Run "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\Tools\VsDevCmd.bat -arch=x86" to initialize Visual Studio environment (or open VS terminal). For community version, run similar command "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Tools\VsDevCmd.bat -arch=x86"
• Create build directory mkdir build;cd build
• Run (to get a Release build)

cmake -G "Visual Studio 16 2019" -DCMAKE_BUILD_TYPE=Rel  ^
    -DENABLE_TESTS=ON ^
    -DNETCDF_PREFIX="C:\OSGeo4W64" ^
    -DHDF5_ROOT="C:\OSGeo4W64" ^
    -DGDAL_DIR="C:\OSGeo4W64" ^
    -DGDAL_LIBRARY="C:\OSGeo4W64\lib\gdal_i.lib" ^
    -DGDAL_INCLUDE_DIR="C:\OSGeo4W64\include" ^
    -DLIBXML2_LIBRARIES="C:\OSGeo4W64\lib\libxml2.lib ^
    -DLIBXML2_INCLUDE_DIR="C:\OSGeo4W64\include\libxml2" ^
    -D EXTERNAL_DRIVER_DHI_DFS=OFF ^
    ..

• Build the executables cmake --build . --config Release
• Adjust the PATH (or copy the mdal.dll to folder test\Release) with command set PATH="C:\OSGeo4W64\bin;<your_build_dir>\build\tool\Release\;<your_build_dir>\build\mdal\Release;%PATH%"
• Run the tests ctest -VV -C Release --exclude-regex mdal_dynamic**

Linux

install dependencies for drivers

sudo apt-get install libgdal-dev libhdf5-dev libnetcdf-dev libxml2-dev

and use cmake to generate build system

mkdir build;cd build
cmake -DCMAKE_BUILD_TYPE=Rel -DENABLE_TESTS=ON ..
make

MacOS

First you need to install qgis-deps dependencies,

To build, create build system with cmake and make sure you use dependencies from qgis-deps and not system ones

mkdir build;cd build
cmake -DCMAKE_PREFIX_PATH=/opt/QGIS/qgis-deps-0.5.3/stage -DCMAKE_BUILD_TYPE=Debug -GNinja -DENABLE_TESTS=ON ../
ninja

External Drivers

see DFSU

Tests

run ctest command in build directory ctest -VV

Code syntax

format code:

cd scripts
./check_all.bash

or use git pre-commit hook

cd MDAL
ln -s ./scripts/mdal_astyle.bash .git/hooks/pre-commit

Windows

On Windows, the code can be formatted using the following :

cd MDAL
powershell -ExecutionPolicy BYPASS scripts\check_all.ps1

mdal's Issues

Support ASC folder

support folder with multiple ASC files, one file - one timestep

add valgrind check to travis

Add support for UGRID file

UGRID: Delft3D Flexible Mesh Suite

Lazy Loading for HDF5 outputs in FLO-2D format

also use mins and maxs arrays for statistics

related to #28

Add mesh calculator

Add HEC-RAS support

Support for dataset groups

Support telemac outputs

Not detecting velocity arrows

See ticket #18 for sample data.
NB: Crayfish does show the velocity arrows.

Support Datasets

• infrastructure for datasets
• support bed elevation
• support reading dat files

Very slow rendering of a small GRIB file

https://t.co/iamnxajML8

I am struggling to render this file. I think it is a problem with NODATA value.

QT dependency ?

I'm aware the current code is just a stub, but it uses QT. Is it really needed ? Plain C++11 with STL is not enough ?

Add write API to be able to create binary dat dataset

travis does not error out on wrong code syntax

astyle is run, but there is no error raised

Add 3Di format

.dat Format not Recognized in QGIS

It appears the loader (MDAL::Loader::load) does not attempt to load the binary or ascii .dat file formats. Thus when QGIS attempts to load a .dat mesh, it errors out and says it's an unrecognized format.

Not possible to open nc file

I have an netcdf file was opening fine in QGIS 3.4 (see the blog post about mdal: https://www.lutraconsulting.co.uk/blog/2018/10/18/mdal/) but the same file can't be opened in QGIS master (with gdal master):
https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc

Unable to open some datasets from ECMWF/Copernicus

see
https://lists.osgeo.org/pipermail/qgis-developer/2018-November/055205.html

Travis Win with secrets

win build is failing:

https://travis-ci.community/t/current-known-issues-please-read-this-before-posting-a-new-topic/264

VM fails to come up if you have secrets
This happens whether they are in Settings, or encrypted in .travis.yml.

add support for XDMF format

Generic format for scientific data.

Support TIF/ASC folder

multiple TIF files in one folder
one TIF - one time

2DM meshes with a gap in node numbering

lutraconsulting/qgis-crayfish-plugin#74
lutraconsulting/qgis-crayfish-plugin#72

Use int in public API

In QGIS, we use QVector, where size() gives int
After #1 we use std::vector where size() gives size_t

Somewhere we need to do a conversion and potentially raise errors, but for now it is ok to just use static_casts to convert to int

Lazy loading for Hec RAS format

it is based on HDF5

related to #29

expose dataset (group) min/max in API

Some formats do store precalculated min/max in the format, so it is way faster to get it directly.

something similar to GDALComputeRasterStatistics

R package for mdal (rmdal)

It will be great to have R package for mdal
CC @mdsumner

Problem with grib2 format

If I have a grib file with grib2 extension, it is not possible to add it to QGIS as mesh layer. But changing the extension to grb, the file can be added as a mesh layer.

Feature Request: Show elevation of mesh when hover over

In the crayfish plugin the elevation of the mesh at cursor-position was shown in the the crayfish-panel.

It would be nice to have a similar feature in MDAL. It would also be possible to use the value tool plugin for that.

Question: c++11 required?

I noticed in the source quite a lot of new/delete calls and factory methods which return raw pointers. Have you given any thought yet to a minimum version of c++ required to build this project?

If it's >= c++11, I'd suggest utilising unique_ptrs everywhere you can in the public api - so that e.g. factory methods always return a unique_ptr and ownership is never in doubt.

Add support for AnuGA output

Support MIKE 21 file format (.dsfu)

.dfsu is our flexible mesh format (triangles and quads)

Modern MIKE models are built in MIKE21 Flexible Mesh (FM) (or MIKE 3 FM) and output dfsu format by default.

format is binary with closed-source implementation, accessible by MIKE SDK

add code coverage report on travis

enable mdalinfo test for windows

Support XMDF format

https://en.wikipedia.org/wiki/XMDF

Support HEC-RAS 5.0.5+ outputs

Netcdf support

It will be good to add support for netcdf:
https://github.com/lutraconsulting/qgis-crayfish-plugin/blob/master/corelib/frmts/crayfish_netcdf.h

https://github.com/lutraconsulting/qgis-crayfish-plugin/blob/master/test/data/NetCDF/indonesia_nc4.nc?raw=true
https://github.com/lutraconsulting/qgis-crayfish-plugin/blob/master/test/data/NetCDF/indonesia_nc3.nc?raw=true

Problem with HDF driver

With gdal master and QGIS master, I can't load xmdf in QGIS any more:

HDF5-DIAG: Error detected in HDF5 (1.10.0-patch1) thread 140439973355776:
  #000: ../../../src/H5Gdeprec.c line 278 in H5Gopen1(): unable to open group
    major: Symbol table
    minor: Can't open object
  #001: ../../../src/H5Gint.c line 289 in H5G__open_name(): group not found
    major: Symbol table
    minor: Object not found
  #002: ../../../src/H5Gloc.c line 430 in H5G_loc_find(): can't find object
    major: Symbol table
    minor: Object not found
  #003: ../../../src/H5Gtraverse.c line 869 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found
  #004: ../../../src/H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed
  #005: ../../../src/H5Gloc.c line 385 in H5G_loc_find_cb(): object 'Difference' doesn't exist
    major: Symbol table
    minor: Object not found

MDAL mesh layer not loading

I acknowledge issue #72.
I ran QGIS 3.4.3 as admin, loaded my 2dm. When I assign extra datasets to mesh (XMDF file), a "BAD ALLOCATION" error is returned. XMDF is 3.3GB.

However, the error does not turn up when the XMDF is about 800 KB.

Problem opening mesh layer in QGIS as user

Trying to open a mesh layer as user fails in QGIS with the following error:

But when you run QGIS as admin, the same layer can be loaded. Note, this happened for a 2dm file.

Support for MIKE (dfs2) format

Would be nice to see support for the mike dfs2 format.

Submit results to CDASH

Support for GRIBB data

Currently GRIBB data can be loaded through gdal, but in reality it is a mesh layer and better to be handled by mdal.

Lazy loading of 2dm mesh

Add FLO-2D format

Lazy loading of XMDF dataset

specially xmdf (hdf5) format datasets should be loaded only when requested.

Read mesh vertices/faces in batch, not one by one

meshes and datasets. To allow for some providers to load data in chunks and not all in-memory.

enable back netcdf gdal test on OSX

gdal2 from homebrew seems to be without netcdf driver (?) and building with all drivers timeouts travis. the netcdf test is not run on OSX for now.

Support for hydro_as-2d files

missing file

When I try to compile MDAL, I receive an error message about the file mdal_config.hpp which is missing in the repository.
Could we comment the include without any risks?
Regards

Reload mesh on refresh

csv and shapefile layers re-read the data from disk when pressing F5. This currently does not work with meshes.