unity-sds / sbg-workflows

example workflows for the sbg sister unity work

Readme

Repository to store the SBG workflows implemented within unity. SBG Workflows consists of:

1. L1B pre-process
2. L1B ISOFIT
3. L2 Spectral Resample
4. L2 Reflectance Correction Resample
5. L2B Fractional Cover
6. L2 Train Estimation (Terestrial Ecology)

The above workflows consist of search (when required), stage-in, process, stage-out, and catalog steps. The inputs and outputs required can be seen in the workflow input sections below.

Additionally, there is an end-to-end wokflow encompassing workflows 1-5 above. the Train Estimation workflow is executed when the outputs of

Workflow Inputs and Outputs

sbg-workflows's Issues

Example Application Package Workflow Wrapper

Create a template repo (or add to the application package template repo) a workflow that can...

1. Search for data
2. run the Application package (stage-in, process, stage-out)
3. catalog the results

With clearly defined places to add inputs, etc.

SBG L2 Train Estimation Application package

SBG trait estimation application package creation
Repo: TBD
Sister implementation: https://github.com/sister-jpl/sister-trait_estimate

Acceptance Criteria:

• algorithm code and required libraries are included in repository environment.yml file
• Application package is built and registered within Docker Hub
• Application package is built and registered within Dock Store
• Application Package CWL is created
• Output Datasets have been defined (to U-DS)
• Example inputs, outputs as expected
• Example deploy and execute commands (unity-py)
• Deployed and tested within WPS-T existing SPS

SBG pre-process application Package

SBG Pre-process step application package creation
Repo: https://github.com/unity-sds/SBG-unity-preprocess
Sister implementation: https://github.com/sister-jpl/sister-preprocess

Acceptance Criteria:

• pre-process code and required libraries are included in repository environment.yml file
• Application package is built and registered within Docker Hub
• Application package is built and registered within Dock Store
• Application Package CWL is created
• Output Datasets have been defined (to U-DS)
• Example inputs, outputs as expected
• Example deploy and execute commands (unity-py)

SBG spectral Resample application package

SBG spectral Resample application package creation
Repo: TBD
Sister implementation: https://github.com/sister-jpl/sister-resample

Acceptance Criteria:

• spectral re-sample code and required libraries are included in repository environment.yml file
• Application package is built and registered within Docker Hub
• Application package is built and registered within Dock Store
• Application Package CWL is created
• Output Datasets have been defined (to U-DS)
• Example inputs, outputs as expected
• Example deploy and execute commands (unity-py)

SBG L2B Fractional Cover application package

SBG L2B Fractional Cover application package creation
Repo: TBD
Sister implementation: https://github.com/sister-jpl/sister-reflect_correct - similar to preprocess

Acceptance Criteria:

• algorithm code and required libraries are included in repository environment.yml file
• Application package is built and registered within Docker Hub
• Application package is built and registered within Dock Store
• Application Package CWL is created
• Output Datasets have been defined (to U-DS)
• Example inputs, outputs as expected
• Example deploy and execute commands (unity-py)

Define required collections for SBG 1 workflows

Define the collections required to run the various applicaiton packages and store their outputs.

Environment agnostic (e.g. we will create them in dev, test, prod, etc) from an automated script.

SBG Workflow 1 End-to-End workflow

Create the end-to-end workflow that chains all processes together.

SBG ISOFIT Application Package

SBG ISOFIT step application package creation
Repo: TBD
Sister implementation: https://github.com/sister-jpl/sister-isofit

Acceptance Criteria:

• ISOFIT code and required libraries and anc/aux files are included in repository / environment.yml file
• Application package is built and registered within Docker Hub
• Application package is built and registered within Dock Store
• Application Package CWL is created
• Output Datasets have been defined (to U-DS)
• Example inputs, outputs as expected
• Example deploy and execute commands (unity-py)

Deploy Workflow 1 to SBG Venue

Deploy Workflow 1 to the SBG venue via wps-t and test for execution completion

Acceptance Criteria

• Can deploy the workflow and associated application pacakges
• Can execute the workflow with configurable parameters
• Execution runs to completion
• Data products are cataloged in U-DS and stored in SBG S3 bucket.

SBG Reflectance Correction Resample application package

SBG Reflectance Correction Resample application package creation
Repo: TBD
Sister implementation: https://github.com/sister-jpl/sister-reflect_correct - similar to preprocess

Acceptance Criteria:

• algorithm code and required libraries are included in repository environment.yml file
• Application package is built and registered within Docker Hub
• Application package is built and registered within Dock Store
• Application Package CWL is created
• Output Datasets have been defined (to U-DS)
• Example inputs, outputs as expected
• Example deploy and execute commands (unity-py)

Workflow Trigger Notebook

Create a notebook that does the following steps:

1. Downloads the fractional cover and reflectance maps
2. Executes the logic behind the algorithm router (github)
3. Execute the L2 specialized algorithm based on the results of number 2.

Validate train estimation workflow against sister generated products

Acceptance Criteria:

• Identify example L1 inputs for validation
• in Jupyter, access existing data and newly generated data
• create a 'difference map' between the two products for various parameters
• display results