nasa-pds / devops Goto Github PK

💡 Description

For example these repositories:
https://github.com/NASA-PDS/registry-harvest-service/blob/main/docker/Dockerfile
https://github.com/NASA-PDS/registry-crawler-service/blob/main/docker/Dockerfile
https://github.com/NASA-PDS/registry-harvest-cli/blob/main/docker/Dockerfile

I was thinking we could use an environment variable instead so that the CICD sets it automatically

For more information on how to populate this new feature request, see the PDS Wiki on User Story Development:

https://github.com/NASA-PDS/nasa-pds.github.io/wiki/Issue-Tracking#user-story-development

Motivation

...so that I can update any of our PyPi packages.

Additional Details

See NASA-PDS/pds-api-client#2 (comment)

Acceptance Criteria

Given a completed update to a PDS Python repo
When I perform a deplyoment to PyPi
Then I expect to be able to successfully update the PyPi package with my PyPi account.

Engineering Details

This should just be something simple we can add to a "Create a New Python Repo" guide. maybe we just add this to the Python template README?

Motivation

...so that I can easily test services and clients against those services

Additional Details

Acceptance Criteria

Given
When I perform
Then I expect

Engineering Details

We want some generic tool / service identified and a process documented for continuous deployment of PDS EN services that are in development. This capability to be possible for services that may be deployed either (or both):

• on prem
• in AWS

things to thing about:

• will this need to be separate from public Github Actions in order to access our internal servers / AWS services?
• how often do we want to do this? on push to master? nightly? weekly?
• we will need to handle some sort of post-deployment capability (e.g. deploy registry + ingest some data)?

💡 Description

Update https://github.com/NASA-PDS/nasa-pds.github.io/wiki/Git-and-Github-Guide#creating-a-new-repo with the info necessary to enable DOIs for a particular repo when applicable.

Follow-on to #14

💡 Description

The database an sqllite file stored at the root of the virtual environment (doi.db).

The following command initialize the database with the PDS's dataCite dois:

pds-doi-init --service datacite --prefix ${PREFIX} --submitter ${SUBMITTER}

Where submitter is the email of the author of the operation and prefix is one of 10.17189, 10.26007, 10.2603 (to be confirmed with @jordanpadams which one is best in this case). We would get the DOIs from the dataCite test database anyway to avoid sharing the production login/password on the staging area and risking them to be used to update the production DOIs.

💪 Motivation

...so that I can develop my product generation software in parallel of the development of the dictionary, seamlessly.

📖 Additional Details

2 tasks foreseen:

• update LDD-CORRAL to handle the snapshot version of the LDD and publish it

• integrate the LDD-CORRAL (https://github.com/nasa-pds/operations/#ldd-corralpy) to the CICD framework so that stable and snapshot releases of the LDD are published:

  • whenever a commit/push is done on the main branch, update the snapshot release
  • whenever a stable release is created (processus to be defined, I don't know what it is currently)

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

First we were thinking we could use Jenkins for the CICD integration. By doing so we would build on what has already been developed for the CD in integration and I&T environment.

Then I realized the final location of the deployment is going to be the web site hosted on AWS. So we might want to use the opportunity of this user story to start to look into continuous deployment on AWS. I have no idea what the would be yet. This ticket would include the required analysis for doing that.

💪 Motivation

...so that I can have an area to run operations procedures in a non-production environment

📖 Additional Details

See architecture diagram and parent Epic with related tickets for more details

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

The automatic image creation recently added to several repositories (registry-api, registry-loader, registry-harvest-service, registry-crawler-service, and registry-harvest-cli) seems to take into account only Dockerfile.

However, some of the repositories use Dockerfile.local and other Dockerfiles. The automatic image creation should figure out which of these to use for stable and unstable workflows.

📖 Additional Details

Follow-on to #3 design

💪 Motivation

...so that I can perform I&T on the services prior to release

📖 Additional Details

See architecture diagram for more details.

• Ceremony to determine release candidates: Sprint Reviews, TRR
• Venue: see diagram

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

💡 Description

💪 Motivation

...so that we don't have to do it manually build and publish the images.

📖 Additional Details

consider the publication on AWS ECR for later

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

The docker images used in the registry pipeline are not updated.

🕵️ Expected behavior

I expected the newest latest images to be pulled every night.

📜 To Reproduce

Go on the jenkins https://pds-jenkins.jpl.nasa.gov/job/Registry/

🖥 Environment Info

No response

📚 Version of Software Used

No response

🩺 Test Data / Additional context

No response

🦄 Related requirements

🦄 #xyz

⚙️ Engineering Details

No response

Currently the NASA-PDS group on Zenodo (what they call a "community") belongs to the GitHub user @nutjob4life, who is then responsible for listing newly-assigned DOIs in the group.

Zenodo says that "having multiple curators per community is an oft-requested feature" but is "currently not supported" and is recommended that "a single account that an organization has shared access to" be assigned the ownership role.

We have such an account: @pdsen-ci.

Please have it take over the NASA-PDS community on Zenodo.

@MJJoyce and @nutjob4life have been struggling over the last 2⅗ weeks with Python Versioneer and we keep running into issues (from mysterious git hashes when there shouldn't be any, to inconsistent handling of annotated/non-annotated tags, to puzzling out the unseen magic that goes into determining a version, to developing one-too-many workarounds in the Roundup Action and pds-github-util just to accommodate Versioneer).

There are some endemic issues to Versioneer as well, including:

• Python 3.6 is the last officially supported Python, but we have standardized on Python 3.9.
• Features like being able to override the version (essential for a Roundup stable release) was requested over two years ago but remains open.
• It still cannot comprehend the now-ubiquitous main branch, using hard-coded master branch to check for divergence.

Other users are running into these as well—@collinss-jpl in particular, where the pds-doi-service has served as the poster case for all that can go wrong with Versioneer.

One famous maxim from Python development (import this) reads:

Explicit is better than implicit.

Versioneer has made things so implicit that we find ourselves at a loss to explain its behavior—and worse, to correct it. We feel we've surpassed the point of diminishing returns with Versioneer and now recommend its excision—with prejudice—from the PDS codebase. Here's something that, perhaps, Maven has done right: just put the version number in pom.xml explicitly where people know how to find it!

This ticket will serve as an epic for the removal from other repositories that the Versioneer has infected.

💡 Description

For components:

• registry-common (no docker image here)
• harvest (no docker image here)
• registry-manager (no docker image here)
• registry-loader (docker image available here)
• registry-api (docker image creation with mvn spring-boot:build-image)

A specific docker image is created from the dev branch and the latest docker image from the other components are used.

To move forward with that, I suggest to:

1. migrate registry-common, harvest and registry-loader into the registry-loader repository as it was planned before.
2. add the githb action on registry-loader
3. add the github action on registry-api

💡 Description

Using the existing terraform scripts in repository:

• registry-api https://github.com/NASA-PDS/registry-api
• registry-sweepers https://github.com/NASA-PDS/registry-sweepers (to be renamed)

For more information on how to populate this new feature request, see the PDS Wiki on User Story Development:

https://github.com/NASA-PDS/nasa-pds.github.io/wiki/Issue-Tracking#user-story-development

Motivation

...so that I can update / access any of our repositories and source code at any time.

Additional Details

We should make sure all of our repos have the following teams with access:

• @pds-software-committers - Write
• @pds-software-pmc - Admin
• @pdsen-operations - Admin

Additionally, not sure if doing this autonomously is worth the effort at this point. Let's just add this to the template repo READMEs or in a wiki for creating a new repo and call it good.

Acceptance Criteria

Given a member of the @pds-software-committers team has an update to push to a PDS EN repo
When I perform a commit and push to a branch
Then I expect to successfully push to a branch on that repo

Given a member of the @pds-software-pmc or @pdsen-operations team has an update to push to a PDS EN repo
When I perform a commit and push to master on that repo
Then I expect to successfully push to master on that repo

💡 Description

For both on prem and AWS services

💡 Description

The objective is to be able to automatically deploy an up to date I&T platform where automated tests have been executed and published on testrail.

For more information on how to populate this new feature request, see the PDS Wiki on User Story Development:

https://github.com/NASA-PDS/nasa-pds.github.io/wiki/Issue-Tracking#user-story-development

Motivation

...so that we can ensure the 99.99% uptime "requirement" we are striving for across all our systems

Additional Details

This goes beyond the systems monitoring the SAs have setup for our machines. That monitoring only ensures the services are up and running ⚙️ (e.g. Tomcat hasn't crashed), but we need some tool or service that makes sure the API actually returns data.

Acceptance Criteria

Given a registry ingestion fails and wipes out the database (or name your favorite service)
When I perform nothing
Then I expect to get a notification that something is wrong with the registry

Engineering Details

We should think of this as some sort of generic "ping" software (or Github Action), that kicks all our services every so often to see if they are awake and functioning properly. At minimum, this should support the following services to start:

• Legacy Keyword Search (e.g. https://pds.nasa.gov/services/search/search?wt=json&q=identifier:%22urn:nasa:pds:context:instrument_host:spacecraft.mars2020)
• PDS API (which will also test the registry)
• DOI Service

💪 Motivation

...so that I can run clients against an API server without needing the expertise to set up an API server

📖 Additional Details

The parent epic ticket as well as the deployment lifecycle diagram provide context and additional details.

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

💡 Description

Because the hosts provided by github do not allow reliable execution of the github actions, we want to deply and execute them on other premises (e.g. AWS)

💡 Description

Take into account the following components:

• web portal
• registry
• nucleus

The priority is on the registry deployment.

Example of deployment strategy is blue/green.

Checked for duplicates

Yes - I've already checked

🧑‍🔬 User Persona(s)

EN team member

💪 Motivation

...so that I can report on the automatic testing.

📖 Additional Details

No response

Acceptance Criteria

Given the jenkins dev staging server, testrail production service
When I perform when I go to the testrail run report https://cae-testrail.jpl.nasa.gov/testrail/index.php?/runs/overview/168
Then I expect to see nighty test reports for the suite PDS Registry API, automated tests (https://cae-testrail.jpl.nasa.gov/testrail/index.php?/suites/view/12824&group_by=cases:section_id&group_order=asc)

⚙️ Engineering Details

Follow the guidelines on https://medium.com/apis-with-valentine/how-to-integrate-testrail-with-postman-newman-api-tests-cc0380998d04

Use test suite in testrail https://cae-testrail.jpl.nasa.gov/testrail/index.php?/suites/view/12824 for API automated testing.

Use test with ID: C1274687

Add the reference in the postman test suite in the registry repository, see https://github.com/NASA-PDS/registry/tree/main/docker/postman

Update the newman configuration for the jenkins pipeline so that jenkins generates a nightly test report on test rail. That will be a single test for now, but we want that to be a starting point for the I&T team to get a full reporting on the postman tests in test rail.

💡 Description

Define the proper revision scheme related to the actual version of the components.
That could be using the version of the components or tags (latest, stable...)

Applies to the following repositories:

For registry:

• registry-api
• registry-sweepers

For nucleus:

• validate
• deep-archive
• registry-loader

Validate that the tasks can be used on the AWS deployments (e.g. en-delta for registry).

• RDD is release-specific documentation
• Do we need software summaries? Should these just be included in the RDD?
• Software Catalog - page of all tools available. very similar to software summary, but I don't care about past builds or that builds are a thing. just want the tools I need right now

💡 Description

Applies to the following repository:

For registry:

• registry-api
• registry-sweepers

For nucleus:

• validate
• deep-archive
• registry-loader

Validate that the docker images work on the AWS deployment (e.g. on en-delta for registry). It is likely that the Dockerfile will require update since the one currently used for registry-api in the test environment (docker compose) and generated by the github action is different that the one used in AWS.

Per the JPL-PDS Open Source Policy:

Each major release of the software must have a persistent identifier such as a Digital Object Identifier (DOI). 

Open Questions

• The policy says "persistent identifiers". DOIs are the default for this outside of the PDS. Is that the route we want to take?
• Apparently this is pretty easily integrated with Github, but, surprise, I don't think it is easily automated. Can we automate? Or is it just a procedural update to the wiki?

💡 Description

Jobs should be triggered by changes on the github repositories

💡 Description

doi service should be on url /api/doi// where version is currently 0.2, but we should have latest instead
registry api should be on /api/search/ where version is currently 1.0 , but we should have latest instead

As part of our "lessons learned", having PDS repositories use the main branch of Roundup Action causes problems—namely, if a feature is implemented or a bug is fixed that introduces a new bug, every repository using main is affected.

Instead, when we have a point in the Roundup development that we consider stable, we tag it with stable. A workflow YAML file can say:

uses: NASA-PDS/roundup-action@stable

to say it prefers that tag of the action and not some other branch—especially not main.

The following repositories are still using the @main branch of the Roundup Action. These should should be transitioned to use @stable:

• api-search-query-lexer (awaiting merge)
• archive-analytics (awaiting merge)
• harvest (awaiting merge)
• pds-api-javalib (awaiting merge)
• pds-api-pythonlib (awaiting merge)
• pds-api-service (awaiting merge)
• pds-deep-archive (awaiting mrege)
• pds-doi-service (awaiting merge)
• pds-registry-app (awaiting merge)
• pds-registry-common (awaiting merge)
• pds-registry-mgr-elastic (awaiting merge)
• pds-template-repo-java (awaiting merge)
• pds-template-repo-python (awaiting merge)
• pds4-information-model (awaiting merge)
• pds4-jparser (awaiting merge)
• pdsen-maven-parent (awaiting merge)
• registry-api-service (awaiting merge)
• supplementer (awaiting merge)

💪 Motivation

...so that I can make sure the integration tests run and so that I update the integration tests to test the feature I am developing.

This will also ease the development process by automating the docker image generation and publication and having that done in github actions and pushed to docker hub from there, rather than doing that manually from a laptop.

📖 Additional Details

Integration tests are tests which requires the deployment of muultiple components, which source code is distributed in different github repositories. We want to aggregate them in application repositories (e.g. registry) which contains the code to deploy the different components from docker images and run the integration test.

I was thinking we could publish docker images with a suffix named after the branch, for example registry-api:1.2.0-SNAPSHOT.{name of the branch}

I don't know what would trigger the generation of the image and the deployment in an integration environment. Every commit/push sounds too often and not needed. In my previous job we were using a slack channel to command a robot which would deploy in the integration environment: deploy {env-name} {application} {branch}

I guess that could be managed in jenkins.

We can discuss the details more during one breakout meeting.

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

💡 Description

Consider an update of the new brokeup component, see NASA-PDS/pds-github-util#73

💡 Description

Ask SAs

💪 Motivation

...so that I can check if that works, so that I can add an application.

📖 Additional Details

We need a documentation to access the jenkins server and check that deployment work and to be able to add new applications.

I would expect this documentation to be around there https://wiki.jpl.nasa.gov/display/PDSEN/Developer+Guides since it is dedicated to the dev team and not available outside JPL.

⚖️ Acceptance Criteria

Given the PDS Engineering node internal documentation
When I perform go to the devlopers guidelines
Then I expect to find a documentation for the jenkins platform and the CD on the DEV STAGING venue

⚙️ Engineering Details

💪 Motivation

...so that I can execute operations procedures in a production environment

📖 Additional Details

See architecture diagram and parent Epic with related tickets for more details

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

💡 Description

For example in this page: https://pds-jenkins.jpl.nasa.gov/job/Registry/356/execution/node/49/log/

💡 Description

@nutjob4life knows how to do that, @sjoshi-jpl has access to NGAP.

Brief Description

Based upon NASA-PDS/template-repo-python#5 and https://github.com/NASA-PDS/pds-template-repo-java, lets:

• cleanup https://github.com/NASA-PDS/pds-template-repo-generic so it contains similar README to NASA-PDS/template-repo-python#5
• update https://github.com/NASA-PDS/pds-template-repo-java with some of the things highlighted here to create a template Java repo that will work out of the box for unstable / stable roundups

See https://github.com/NASA-PDS/validate or https://github.com/NASA-PDS/registry-api-service for example java repos

Deliverable

• Link to working execution of an unstable release
• Link to working execution of an stable release

Checked for duplicates

Yes - I've already checked

🧑‍🔬 User Persona(s)

other developers or testers mostly.

💪 Motivation

...so that I can deploy PDS EN docker images on my macbook using a M1 chip.

📖 Additional Details

This need to be implemented in the CICD.

Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

No response

See sub-tasks for next steps

💡 Description

For deployment on NGAP now.

Components to terraform:

• AWS managed OpenSearch (includes Kibana), with registry-manager creation of the schema
• registry-api
• registry-sweeper
• scalable harvest (optional)

💡 Description

Sometimes the version of the application is hardcoded in the Dockerfile file, which requires to update it (and not forget to do so) whenever a new stable release is made.

We would like the application to be an argument of the docker build command that roundup can manage, so to simplify the life of the release manager.

💡 Description

The GitHub Actions Blog reminds us that using ::set-output (and ::save-state) is deprecated for security reasons and to use echo "NAME=VALUE" >> $GITHUB_OUTPUT instead. (I don't think we're using ::save-state at all.)

The repositories that may affect includes:

• registry-api
• registry-crawler-service
• registry-harvest-cli
• registry-harvest-service