geoscienceaustralia / dea-waterbodies

DEA Waterbodies is a product that maps and monitors open waterbodies across Australia. Once a polygon set has been generated corresponding to open waterbodies, each waterbody is tracked over time to record the change in wet surface area over time.

Digital Earth Australia Waterbodies

License: The code in this repository is licensed under the Apache License, Version 2.0. Digital Earth Australia data is licensed under the Creative Commons by Attribution 4.0 license.

Contact: If you need assistance with any of the Jupyter Notebooks or Python code in this repository, please post a question on the Open Data Cube Slack channel. If you would like to report an issue with this repo, or suggest feature requests, you can open an issue on this repository. Non-technical questions about Digital Earth Australia Waterbodies can be sent to [email protected].

Citing Digital Earth Australia Waterbodies:

Krause, Claire E.; Newey, Vanessa; Alger, Matthew J.; Lymburner, Leo. 2021. "Mapping and Monitoring the Multi-Decadal Dynamics of Australia’s Open Waterbodies Using Landsat" Remote Sens. 13, no. 8: 1437. https://doi.org/10.3390/rs13081437

---

Up to date information about the extent and location of surface water provides all Australians with a common understanding of this valuable and increasingly scarce resource. Water detection algorithms are now being routinely applied to continental and global archives of satellite imagery. However, water resource management decisions typically take place at the waterbody rather than pixel scale.

This repository presents a workflow for generating polygons of persistent waterbodies from Landsat observations, enabling improved monitoring and management of water assets across Australia. We use Digital Earth Australia’s (DEA) Water Observations from Space (WOfS) water classifier, which provides a water classified output for every available Landsat scene, to determine the spatial locations and extents of waterbodies across Australia. DEA Waterbodies uses Geoscience Australia’s archive of over 30 years of Landsat satellite imagery to identify where almost 300,000 waterbodies are in the Australian landscape.

Digital Earth Australia Waterbodies workflow

Each polygon was then used to generate a time series of WOfS, providing a history of the change in the wet surface area of each waterbody every ~16 days since 1987.

Digital Earth Australia Waterbodies. Waterbody polygons mapped by this product are shown in blue. There are almost 300,000 across Australia.

DEA Waterbodies supports users to understand and manage water across Australia. DEA Waterbodies provides new insights into local through to national-scale surface water spatio-temporal dynamics by enabling the monitoring of important landscape features such as lakes and dams, improving our ability to use earth observation data to make meaningful decisions. It can be used to gain insights into the severity and spatial distribution of drought, or identify potential water sources for aerial firefighting during bushfires.

For more information about the DEA Waterbodies product, including instructions for accessing the product, frequently asked questions and data download links, see the Digital Earth Australia website.

Installation

DEA Waterbodies has some requirements which can be installed with pip:

pip install --extra-index-url="https://packages.dea.ga.gov.au" -r requirements.txt

Once you have installed the requirements for DEA Waterbodies, install the module locally:

pip install -e .

This command installs an editable version of the module in the current location.

A command line interface is available for generating wet area time series for a given shapefile. You can call the help for this interface from the command line using:

waterbodies-ts --help

dea-waterbodies's Issues

Use setuptools_scm for versioning

#13

Why do Lake Gordon, Lake Pedder, and Lake Will, TAS, have incorrect polygons?

These are two lakes with OK looking WOfS annual summaries, but broken polygons in DEA Waterbodies.

Add coverage metrics via CodeCov

CodeCov has had a bad rap lately for security, but it still offers one of the best UI's for FOSS coverage metrics. Use pytest-cov to generate coverage upload to codecov.

Please see here for sample : https://github.com/opendatacube/datacube-ows/blob/b84eceeb7f97c1b91c95e6413d8df2c6fe9b82b8/.github/workflows/test.yml#L64

Use fsspec to unify S3 and local IO

I know this is already well underway, but making an issue to keep track of it!

Allocate chunks with area of bbox instead of area of polygon

make_chunks ignores filter_state etc

Tests fail if you have a datacube instance that has too much data indexed

i.e. test_make_csv and friends assume that you have only a handful of scenes indexed

Replace Kati Thanda with Geofabric polygon

...or some other source of truth. We should examine the difference between the two as well. I think this would make it much faster to run Kati Thanda since the polygon would be less hugely complex, and we'd capture the actual behaviour of the lake regardless of its fill in the last 30 years.

Improve logging

Logging should record more of what is going on so we can more easily debug stuff on Airflow.

Add help to CLI for waterbodies

From line 166 in make_time_series.py using click.

Cache package dependencies in Github Action

The slowest part of automatically testing at the moment by far is the installation of dependencies, notably dea-notebooks. We should cache this to improve testing runtimes.

Refactor waterbody time series functions to accept GeoDataFrames

Makes testing and code reuse a bunch easier.

Write local setup script

In particular ./setup_test_datacube.sh doesn't work without the right environment variables

CLI `--all` option not enabled

See https://github.com/GeoscienceAustralia/dea-waterbodies/actions/runs/1054591615 for errors that appear when --all is chosen instead of --some.

Add versioning information to repo documentation

The following decisions on versioning were made by project members. We need to add this to the repo documentation somewhere:

• WB polygons and csvs to be named by etc
• Versioning
  • Semantic versioning – major, minor and patch
  • Docker will be updated for all updates (m, m, p)
  • Patch – backwards compatible (e.g. add a new column, new data but the existing data doesn’t change, minor debugging)
  • Minor – incremental breaking changes (e.g. restructuring folders, deleting data, modifying schema)
  • Major – considerable updates we want to promote – changes + comms (e.g. any time we majorly change the polygons, if there’s a chance we can change analysis outputs).
• Single branch in DEA WB github page tagged with releases

Make a faster method for pushing messages to the SQS queues

Currently this is done in Airflow with xargs but it'd be good to have a Python-based way to do it to get some scripting out of there.

Aggregate polygons then filter by number of observations

Currently, the code filters pixels on the number of valid observations prior to vectorising.

By changing the order of the filtering, it allows polygons in Tas, which are always wet but infrequently observed, to be better captured.

The impact on other locations where invalid pixels may end up being included in waterbody polygons needs to be explored.

Combine Waterbodies with Geofabric to attribute names etc

The BoM's Geofabric dataset contains names and attributes for waterbodies mapped within it. It would be a useful enhancement of DEA Waterbodies to have these attributes.
A spatial join may help with combining attributes, but it will also likely be very very messy.

Check that the geodata 100k coastline mask behaves OK

This version of the polygon generation uses a datacube-indexed coastline mask instead of @CEKrause's custom coastline mask. We should check that the results are sufficiently similar and that the coastline mask behaves correctly in the ocean.

Make geohash coordinates lay inside polygons

At the moment, geohashes are applied to the centroids of polygons. If a polygon is 'U'-shaped, then the centroid for the polygon will occur outside of the polygon itself, and the geohash will not make it clear which polygon it refers to.

Change the way the geohash is applied and apply it to the centroid of the polygon AND inside the polygon.

Some waterbodies have STATE=NULL

Time series generation fails if there are multipolygons

PROCESSED_FILE maybe doesn't work

I think an if needs negating.

Fix or silence all the warnings

Make a way to run tests locally (without destroying the local datacube environment)