podaac / swodlr Goto Github PK

After some initial discussions about the proposed architecture (https://github.com/podaac/swodlr/wiki/Architecture), implement a simple API for working with the domain objects discussed.

Acceptance Criteria:

• User should be able to create an L2 Raster Product object
• User can view a L2 Raster Product object that was previously created

Tool which performs realistic load simulation for end-to-end test of the raster-create process

https://hysds-core.atlassian.net/jira/software/c/projects/HC/issues/HC-474

Create HySDS job + user rule that can be independently imported into HySDS to provide SWODLR with event data.

AC:

• Automated test runs successfully that logs in a test user with EDL

Initial implementation, linting, testing, and CI/CD

• Update database schema to provide S3 URI storage for generated products
• Update ProductHistory to provide requested timestamp in GraphQL schema
• Bucket to TEA endpoint mapping in Spring configurations
• S3 bucket URI to TEA URI converter
• Unify pagination strategy across all searchable objects
• Implement frontend redirect from OAuth2 flow

Expand ingest-to-sds to be step function with sqs as input
Group other lambdas into step functions

It is likely this will be required by NASA / Earthdata Cloud in the near future. We should get in front of it and enable encryption on the RDS database now since we're still early in development.

Collected metrics should include:

• time received
• time job submitted to the SDS
• time job completes

• Add notify_update to the post-evaluate step in the step function to provide more visibility into generation
• Create the publish_data lambda and add to step function
• Unit tests for publish_data
• Fix CI/CD pipeline variable

Implement CI/CD pipelines via GitHub Actions

First pass AC (feel free to suggest a change):

• Pipeline performs linting/unit tests on all branches
• Pipeline performs .jar and Docker container builds for API
• Pipeline pushes generated artifacts to GitHub Repository
• Pipeline automatically increments version number based on branch (main/develop/*), and increment develop branch based on release; likely to be handled by bump2version
• Pipeline generates tags based on branch (main/develop/release); likely to be handled by bump2version

Fix API service to provide proper CORS headers/cookie options depending on environment deployed in. SIT environment should relax security policies to allow for local frontend development

Implement pre-defined configurations for users to select:

(1) output_granule_extent_flag (e.g., 0)
(2) raster_resolution (e.g., 100, 200)
(3) output_sampling_grid_type (e.g., utm)
(4) utm_zone_adjust (e.g., 0)
(5) mgrs_band_adjust (e.g., 0)

User provides Cycle/Pass/Scene IDs & select from above pre-defined configurations provided by our API.

Implement logic that allow user to submit Cycle/Pass/Scene IDs & pre-defined configurations as inputs

In v1 of swodlr UI, an user puts in cycle/pass/sceneID to submit a request to generate customized L2 raster products. The scene ID can be found from the SDP SWOT_L2_HR_Raster granule in its file naming convention, iso metadata, CMR, or is calculated using an equation.

The code that calculates the scene ID via an equation is already developed in the backend service. Create a convenient tool that leverage the existing backend code to provide the end user the scene IDs using the equation.

Modify the GQL schema to better differentiate the raster product querying and creation operations. query.l2RasterProduct should remain the same, but provide documentation that if the product does not exist, it will not be created. mutation.queryOrCreateL2RasterProduct should provide the same functionality as query.l2RasterProduct, but in addition, if a product requested does not exist, it will initiate the generation of the product

This differentiation should cover the two base use cases for Swodlr

Integrate raster-create calls into the API via SQS

Not urgent as we are still in development but I want to track this because we need to fix it before public deployment.

The default error page currently includes stack traces and info that we do not want to be publicly available when the service is published.

Need to create a user-friendly error page that indicates an error occurred. The error page should include a unique error code identifier so that service operators/developers can quickly search application logs using the unique error code to find the relevant stack trace.

https://github.com/podaac/swodlr-async-update

Create tests for the API using an in memory database as storage.

A few test cases that should be present:

• Given product does not exist, create new raster product. Validate new product exists
• Given product exists, lookup raster product. Validate product returned
• Given product exists, lookup status of raster product. Validate correct status returned
• Given product does not exist, lookup raster product. Validate error returned indicating product not found

Implement user notification via email when requested custom raster product is ready for download

A/C: Demonstrate user get an email notification to download when products are generated

• 23.3.1 (sprint 1): cleanup/error handling
  • #61
  • #62
• 23.3.2 (sprint 2): #63
• 23.3.3 (sprint 3): #64
• 23.3.4 (sprint 4):
  • #65
  • #66

• Allow for incoming CNM-R messages without a file extension in the filename to be ingested
• Update tool to mark files as type data per CNM spec

Implement API to query existing metadata in CMR to get a preliminary set of L2 PIXC & PIXCVec products

Ensure that the pipeline works from I&A notification to SDS ingestion.

Progress: Mostly complete; I&A needs to work on bucket permissions

• Initialize new Spring Boot project using JDK 17 and Spring 5.3
• Build project using Gradle 7.5

Just need the skeleton of the project for now

This service is going to have to connect to an RDS instance to store state. Need to setup an RDS instance using terraform.

AC:

• Terraform scripts written and can be manually applied to SIT environment
• RDS instance is created with 2 users: an administrator with ownership permissions and an application user with permissions limited to a specific schema to be used for the application.