actinia-org / actinia-stac-plugin

Plugin for actinia to read STAC catalogs and retrieve data for processing

actinia-stac-plugin

You can run actinia-stac-plugin as actinia-core plugin.

Installation

For installation or DEV setup, see docker/README.md.

DEV notes:

Build

insprired by https://hynek.me/articles/sharing-your-labor-of-love-pypi-quick-and-dirty/

to create a shippable wheel, run

pip3 install --upgrade pip pep517
python3 -m pep517.build .

Versioning:

https://semver.org/ (MAJOR.MINOR.PATCH)

Logging:

in any module, import from actinia_stac_plugin.resources.logging import log and call logger with log.info("my info i want to log")

actinia-stac-plugin's Issues

[Read] Analyse importer values and prepare process (chain piece)

• actinia is responsible for reading in the needed collections, applying filters (space + time) to collect needed items and create the process chain for GRASS GIS
• r.in.gdal / r.import will be used for all imports. actinia will pass information (data location, bands, extent, ...) -> the used location can be found out by the used endpoint.
• single scenes are registered directly as STRDS
• prepare for other filters but implementation is not part of this issue

[Discover] Register - Create endpoint HTTP POST `/stac` to register external STAC collections

• For the prototype state, "stac-url", "stac-instance-id" and "stac-collection-id" need to be provided by the user in the POST body.
• the URL needs to be a link to collection
• as a convention, the stac-collection-id must be the original ID of the collection.

[Discover] Write tests

[Discover] Register - Read collection id from original collection and remove option from POST body

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency pyscaffold to >=3.3,<3.4
• Update dependency pyscaffold to v4

Detected dependencies

dockerfile

docker/actinia-stac-tests/Dockerfile

github-actions

.github/workflows/linting.yml

.github/workflows/python-publish.yml

• actions/checkout v4@692973e3d937129bcbf40652eb9f2f61becf3332
• actions/setup-python v5
• softprops/action-gh-release v2

.github/workflows/test.yml

• actions/checkout v4@692973e3d937129bcbf40652eb9f2f61becf3332
• docker/setup-buildx-action v3
• docker/build-push-action v6

pep621

pyproject.toml

• setuptools >=40.6.0

pip_setup

setup.py

• pyscaffold >=3.0a0,<3.1a0
• stac-validator >=2.2.0

setup-cfg

setup.cfg

---

• Check this box to trigger a request for Renovate to run again on this repository

[COMMENT] Set POST methods in `stac/instances` and `stac/collections` for the addition of instances and collections respectivatelly

The POST method to add new instances and new collection At this moment are located in the /stac endpoint. To keep a logic they might be located in stac/instances and stac/collections.

[Discover] Register - Cache JSON content of collection

Store the JSON response of a collection in the redis database to enable the endpoint which lists all collections to return many collections at once without the need to create multiple HTTP requests.

remove `__pycache__` folders?

There are __pycache__ folders inside the folders, they should most probably be removed because they are generated when installing the plugin?

[Discover] Register - Store needed information in redis database

• store stac-url, stac-instance-id, original collection id and openeoID (stac.<stac-instance-id>.rastercube.<stac-collection-id>)

[Result] Write out geodata result to configurable directory

This should already be the case but please double check if it works as expected.

Check cards in Discover Board

In the board "Discover" for after the prototype, multiple cards were created. They need to be transformed into tickets if implementation should follow and prioritized.

[Discover] List - Create endpoint `/stac/instances`

returns a list of all already registered STAC instances

[Read] Add type `stac` to actinia importer

Piece of process chain should look similar like this:

        {
            "id": "importer_1",
            "module": "importer",
            "inputs": [
                {
                    "import_descr": {
                        "source": "stac.example-instance-id.rastercube.example-collection-id",
                        "type": "stac",
                        "semantic_label": "red",
                        "extent": {
                            "spatial": {
                                "bbox":[[-180,-90,180,90]]
                            },
                             temporal":{"interval":[[
                                   "2021-09-09", "2021-09-12"
                             ]]}
                         },
                         "filter": {...}
                    },
                    "param": "map",
                    "value": "example-red"
                }
            ]
        },

• Why "semantic_label" ? -> OSGeo/grass#1868
• Why "red" ? See https://github.com/stac-extensions/eo/#common-band-names - Although the names are given by the STAC catalog itself and don't always follow the common band names guideline
• Why extent? -> https://openeo.org/documentation/1.0/developers/api/reference.html#operation/describe-collection
• Why filter? -> More filter options need to be implemented step by step without breaking the API all the time, so they should be collected in a single object or array.

Currently there is the parameter sentinel_band available in the importer. Although it would make sense to also use semantic_label for it, we don't touch it for now.

The goal of this ticket is not a full implementation to make this PC work but only a preparation to accept the PC, acknowledge the type stac and prepare for semantic_label, extent and filter. When using type stac in a process chain, actinia shouldn't start processing but return a message like "STAC import is coming soon" or similar.

Hint: the importer part of a process chain can also be used directly with a module, e.g.

    {
      "module": "g.region",
      "id": "g_region_1",
      "inputs": [
        {
          "import_descr": {
            "source": "https://storage.googleapis.com/graas-geodata/elev_ned_30m.tif",
            "type": "raster"
          },
          "param": "raster",
          "value": "elev_ned_30m_new"
        }
      ],
      "flags": "p"
    },

I suspect that this will work out of the box as well (as same code is executed) but it should be tested as well.

Also update API docs accordingly and prepare tests.

[Discover] Move HTTP DELETE methods from <resources> endpoint to <resources/id>

Support STAC collections as processing input via openeo-grassgis-driver

The openeo-grassgis-driver is already able to list the collections from /stac/collections within its own /collections endpoint, so they can be theoretically used for processing (see Open-EO/openeo-grassgis-driver#124).
Now that the import of the real data for processing within actinia-core is implemented, the whole processing workflow can be prepared and tested.

[Discover] List - Create endpoint `/stac/collections`

• returns a list of all already registered STAC collections (with description, temporal & spatial extent, ...)
• It should return all information needed by openEO, eg https://openeo-loose.mundialis.de/api/v1.0/collections

[Discover] Register data from inside GRASS GIS

Already possible via exporter, but not all data are automatically registered. See also https://github.com/mundialis/actinia-stac-plugin/wiki/1-Discover#grassdb-in-stac

[Discover] Register - Support automated recognition of ressource

The stac-url can be any URL contained in a STAC catalog and actinia finds out, which type it is and registers collection(s).

Decision:

• for item => superior collection
• for catalog => follow child links 1x, if they are a collection, it will be registered
• for collection => collection

in any case, the collection_id will be read and stored

How to find out type?

• Item = type feature, from 1.0.0 type collection / catalog possible as well
• before 1.0.0: if licenes + extent exists => collection, else catalog
• see best-practices.md#how-to-differentiate-stac-files

[Discover] Register - support naming convention

https://github.com/mundialis/actinia-stac-plugin/wiki/1-Discover#naming-convention

[Result] Register result in STAC catalog

• Requirement is that this can be done simply via an HTTP POST command containing the STAC json
• This could be triggered with new parameter of exporter module
• A different type of the webhook could be used for this. The postbody needs to be adjusted containing the STAC item and not the actinia resource will be done in separate task

[Discover] Create API docs

[Discover] Add config option for initial STAC Catalog references

Currently when no "defaultStac" intance exists, the plugin harvests the two collections:

• https://earth-search.aws.element84.com/v0/collections/landsat-8-l1-c1 and
• "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a

This would be nice to be configurable.

[Read] Download to tmp folder / resource storage

The data should be downloaded to tmp folder / resource storage, where data will be deleted after processing / can be deleted manually -> look at other import options like sentinel

Also keep in mind that another option is mounting of data where actinia will not use the URL to download the resource but import directly from file path.

[Discover] List - Create endpoint `/stac/collections/<stac_collection_id>`

Returns a single collection (with description, temporal & spatial extent, ...)

For now, HTTP request to collection is wanted to see if cache (from reponse of all collections) is still valid.

This might play a role for the decision how to clean cache later.

Check license in file headers

Fixing the Headers with the wrong Licence

[Discover] Register - Support relative links and replace URLs in stored redis entry

[Discover] Register - Decide how to clear/update cache of collection JSON

When harvesting a STAC collection, the entry is stored in redis. There is no mechanism to update it.

No high priority as the cache can be reloaded by doing the same HTTP POST again and the cache will be overwritten.

[Discover] List - Create endpoint `/stac/instances/<stac_instance_id>`

returns a list of all already registered STAC collections from a certain instance

Add pipeline for linting

For each commit, a pipeline should be triggered to look at the PEP8 style guide rules. If they are not considered well, pipeline should fail. If there are valid reasons, a .flake8 file can be used to ignore certain violations.

A running example can be seen here

[COMMENT] IDs redundance in the endpoint `stac/collections`

The current endpoint list all collections stored in Redis regardless of the instance. The comment was given due to the fact that in this listing might happen that two or more COLLECTIONS provided from different endpoints might have the same ID. The suggestion was to create a prefix "index" that works as identifier for the providers and the STAC read.

Highlights:

• The ID inside each STAC Collection is given by the provider, modifying them might not be a good approach.
• The separation by the identifies in the stac/collections endpoint might work, but the effects in the reading in the OpenEO driver must be considered.
• The concept of intances was not clear and maybe we can think about it relevance inside actinia and OpenEO