cgi-estonia-space / alus

GPU accelerated earth observation data processors

ALUS

Collection of processing operators/routines defined by software - plural of ALU(Arithmetic Logic Unit)
or
ALU for Space/Surveillance etc.

A software project that targets to utilize Nvidia GPUs for processing earth observation data (faster).
Kickstart of this project was funded through ESA's EOEP programme
Current development is funded through ESA's GSTP programme

Developed by CGI Estonia.

Quick performance overview

For further comprehensive evaluation see Wiki

Out of the box usage

Verified releases can be downloaded from - https://github.com/cgi-estonia-space/ALUs/releases/ One can download docker image with all of the needed dependencies from dockerhub or simply docker pull cgialus/alus-devel

Executing

Each algorithm is a separate executable. Currently available ones are (more info and usage in parenthesis):

• Sentinel 1 coherence estimation routine - alus-coh (README)
• Sentinel 1 coherence estimation timeline generation - alus-coht (README)
• Sentinel 1 calibration routine - alus-cal (README)
• Sentinel 2 and other raster resample and tiling - alus-resa (README)
• Gabor feature extraction - alus-gfe (README)
• PALSAR level 0 focuser - alus-palsar-focus (README)

When building separately, these are located at <build_dir>/alus_package

Update PATH environment variable in order to execute everywhere:
export PATH=$PATH:/path/to/<alus_package>

See --help for specific arguments/parameters how to invoke processing. For more information see detailed explanation of Sentinel 1 processors' processing arguments.

Docker example

NVIDIA driver and NVIDIA Container Toolkit must be installed together with docker.

docker pull cgialus/alus-devel:latest
docker run -t -d --gpus all --name alus_container cgialus/alus-devel
docker exec -t alus_container mkdir /root/alus
docker cp <latest build tar archive> alus_container:/root/alus/
docker exec -t alus_container bash -c "tar -xzf /root/alus/*.tar.gz -C /root/alus/"
# Use docker cp to transfer all the input datasets, auxiliary data, then either
docker exec -t alus_container bash -c "cd /root/alus; ./alus-<alg> ...."
# Or connect to shell
docker exec -it alus_container /bin/bash
# Running coherence estimation routine example
./alus-coh -r S1A_IW_SLC__1SDV_20200724T034334_20200724T034401_033591_03E49D_96AA.SAFE \
-s S1A_IW_SLC__1SDV_20200805T034334_20200805T034401_033766_03E9F9_52F6.SAFE \
-o /tmp/ -p VV --orbit_dir <orbit files location> --sw IW1 --dem srtm_43_06.tif --dem srtm_44_06.tif
# Running calibration routine example
./alus-cal -i S1A_IW_SLC__1SDV_20180815T154813_20180815T154840_023259_028747_4563.SAFE \
-o /tmp/alus_S1A_IW_SLC__1SDV_20180815T154813_20180815T154840_023259_028747_4563_Calib_tc.tif \
--sw IW1 -p VV --type beta --dem srtm_42_01.tif

Dependencies

Setup of dependencies

Building

cmake . -Bbuild
cd build
make -j8

Jupyter Notebook

There is a Jupyter Notebook located at jupyter-notebook folder with a user-friendly interface and automated auxiliary file downloads. It can be used in conjunction with binaries to easily execute code. Read the instructions.

Minimum/Recommended requirements

For specific, check each processor's README.

Below are rough figures:

• NVIDIA GPU device compute capability 6.0 (Pascal) or higher
• 2(minimum)/4(recommended) GB of available device memory (some ALUs can manage with less)
• High speed (NVMe) SSD to benefit from the computation speedups
• 4 GB of extra RAM to enable better caching/input-output (GDAL raster IO)

Contributing

Contribution guidelines

License

GNU GPLv3

Release notes

Binary downloads

alus's Issues

Github actions

Duplicate the bitbucket pipeline procedures in Github actions.
Use a CGI corporation allocated AWS account for that.
Add Priit's access too.

License change

• Processor specific
• Make sure that no SNAP ports are not included or linked
• Currently change would be introduced for alus-palsar, alus-gabor and alus-resa
• Apache will be the choice (MIT like but lawyers and big organizatons like it more)

Create version 1.6 release

• Basic S1 metadata
• Jupyter notebook updates (also do not forget to update the docker image)
• polynomial fit for each burst
• GRD support ?!

alus-sar-segment "no matching noise values from noise vectors for azimuth range"

Following error statement is printed:
There were no matching noise values from noise vectors for azimuth range [8535.158366, 8535.158376]. Tile properties x:8192 y:0 w:2048 h:2048

When trying to process the following dataset - S1A_IW_GRDH_1SDV_20230515T034801_20230515T034830_048539_05D6A6_D84E

It could be too strict implementation or a bad dataset. Compare with SNAP behavior.

Cloud Optimized GeoTIFF support for ALUs

• Currently only for coherence
• https://gdal.org/drivers/raster/cog.html
• Add checks that if COG is available, such a format is used instead of default GeoTIFF

OR

Format after processing using GDAL

Coherence processing without orbit files (JRC-5)

Tested, when commenting out ApplyOrbitFileOp::Initialize() contents. Need to add argument handling and logical handling of this situation though. No coordinate and/or raster size differences. Only the band data.
Need to communicate to the users (in documentation) if this is implemented.

• Add an E2E test

Copernicus DEM 30 support (JRC-1)

Check BaseElevationModel.java from SNAP project. Everyhting should start to unwind there.
So for example based on SNAP structure:

• ElevationModel.java interface (there are APIs that are needed during processing)

• Base class for inheritance is BaseElevationModel.java

• Then comes CopernicusElevationModel.java for example that inherits from previous

• Then there are BaseElevationTile and ElevationTile

• First lets refactor the current SRTM3 support and then test if the results remain the same, then lets implement support for others.

Bonus:
Would be nice if we'd get rid of continous resampling as done in GetElevation(), if this would be done once on GPU, things would be much faster.
Following stuff:

• snapengine::bilinearinterpolation::ComputeIndex()
• snapengine::bilinearinterpolation::Resample()

Event logging for CREODIAS (CF-9)

From CF-REQ-PGAAS Requirements_for_new_processor.pdf chapter 2.6 Event logging

The processor shall be equipped with event and error logging. Preferably the logging system should allow changing
the type of logged events (DEBUG, INFO, ERROR). [REQ-LOG-1]
Logs should be generated in structured logging format. [REQ-LOG-2]
Example message:

{ "time": "2010-01-01 12:34:56.0000", "level": "ERROR", "message": "hello, world", "elapsed": 10,
"unit": "s" }

Messages should be generated in the proposed format due to external systems receiving, parsing and indexing logs.
If the supplied processor includes an error logging system then the documentation for this processor must include
additional information regarding the details of the system. The documentation must include additional information:
where log files are created, a description of the format of created logs, a description of typical logged messages, how
to control the level of logged events, whether a log archiving mechanism is included, and whether a mechanism for
limiting the space available for logging is included. [REQ-DOC-6]

• Create a secret argument that switches the log format output to match the requirement. Argument name: '--log_format'. It should be called with a value 'creodias'
• Create boost::log formatter accordingly (see branch creodias_log)
• Add argument to alus::log::Initialize to specify format with enum. Currently then DEFAULT, CREODIAS
• Also tune the SetLevel() based on format. CREODIAS requires 3 levels - INFO, DEBUG, ERROR

Requirement CF9

Update thermal noise removal indexing

senbox-org/microwave-toolbox@38418ee

S1 GRD thermal noise removal error

Hi,
I've tested your S1 GRD calibration routine which in general works well. But in some cases, I've got an error during the thermal noise removal procedure:

[2023-06-11 14:02:20.028403] [0x00007f96c7ab0000] [info]    Processing parameters:
Input product - 
/tmp/S1B_IW_GRDH_1SDV_20190405T230644_20190405T230711_015678_01D696_7102.SAFE
Subswath -
Polarisation - VV
Calibration type - sigma
First burst index - 0
Last burst index - 0
AOI -
Write intermediate files - NO

[2023-06-11 14:02:20.805075] [0x00007f96c7ab0000] [info]    Using 'NVIDIA GeForce GTX 1080 Ti' device nr 0 for calculations
[2023-06-11 14:02:20.828726] [0x00007f96c7ab0000] [info]    Target bands count: 1
[2023-06-11 14:02:20.828740] [0x00007f96c7ab0000] [info]    Processing band Amplitude_VV_VV

[2023-06-11 14:02:34.663293] [0x00007f96c7ab0000] [error]   Caught an exception
There were no matching noise values from noise vectors for azimuth range [7034.963325, 7034.963331]. Tile properties x:16384 y:16384 w:2048 h:1756
[2023-06-11 14:02:34.663311] [0x00007f96c7ab0000] [error]   Exiting.

Thank you!
Bjoern

Final presentation assembly

A dedicated server that has the ALUs binaries.
Can run live processing.
Results export to SentinelHub?!

Error with S1 IPF < 2.9

Hi,
it seems that ALUs is not working with S1 imagery before 2018-03-13 12:00:00 (IPF < 2.9).

Found 6 crossing COP-DEM tiles
[2023-06-11 14:44:05.101830] [0x00007f9a5bb9d000] [info]    Processing parameters:
Input product - 
/tmp/S1A_IW_GRDH_1SDV_20180304T231511_20180304T231540_020872_023CD1_3037.SAFE
Subswath -
Polarisation - VV
Calibration type - sigma
First burst index - 0
Last burst index - 0
AOI -
Write intermediate files - NO

[2023-06-11 14:44:05.463604] [0x00007f9a5bb9d000] [info]    Using 'NVIDIA GeForce GTX 1080 Ti' device nr 0 for calculations
Segmentation fault (core dumped)

Are you aware of that issue?

Best
Bjoern

Local UTM projection for SAR products (JRC-8 GEN-1)

• Currently ALUs only supports WGS84 projection by default. At least UTM projection (by zone) is needed. This concerns SAR processors only.
• Currently resampling supports it, need to refactor and create a separate unit that is used for this. Terrain correction by default uses degrees. TiePointGeocoding might need enhanced or simply converting from WGS84 to whatever projection then. Also meters are calculated, sophistication may be introduced when target geocoding system uses meters instead. Also for sure needs confirmation against SNAP. Special nightly added.

GRD support for calibration routine

JRC-4

Resample Copernicus DEM tiles to unified width

There is an issue with SNAP inherited logic - https://forum.step.esa.int/t/copernicus-dem-complications-when-coregistering-s1/38659/2

Biggest issue is coregistering reference and secondary scenes' pixels.

Solution for this would be to resample all the tiles to the same width.

Basic metadata addition to output products of S1 routines

Spatio Temporal Asset Catalog (STAC) compatible output

Jupyter notebook docker (GEN-5)

alus-resa bug accessing S2 bands

Shapefile AOI support (GEN-4 ?JRC-10?)

This includes alus-coh, alus-coht, alus-cal, alus-resa

Polynomial fit for each burst in flat earth substraction for coherence (GEN-3)

Enhance validation scripts

• One that runs SNAP GPT (arguments for which DEM to use and output file location/name)
• The one that runs SNAP GPT and ALUs AND runs rastcomp to compare

Merge gaps between subswaths for specific calibration routine data

Scene in question - S1A_IW_SLC__1SDV_20210722T005537_20210722T005604_038883_049695_2E58.SAFE

• Run the same on SNAP
• Run the same without AOI, all 3 subswaths included
• Run the same for coherence routine