ploigos / ploigos-step-runner

Ploigos Step Runner (PSR) implemented as a Python library.

ploigos-step-runner

Ploigos Step Runner (PSR) is a middleware tool to execute Continuous Integration (CI) pipeline steps. PSR is not a pipeline! It is a CLI tool called in each stage of a pipeline as a command: psr -s package -c psr.yaml.

PSR is tech stack agnostic and runs on workflow runners like Jenkins, Tekton, GitHub Actions, GitLab CI, or any other workflow runner that can execute commands.

PSR abstracts the implementation of a step from a specific product or solution. A few common steps most pipelines implement are unit testing, packaging, and deploying. The implementation of these steps differs between a Java and JavaScript application. PSR abstracts implementation from the pipeline definition, allowing the same pipeline definition to be reused for all applications, regardless of the application language and framework.

Each PSR step has one or more Step Implementers that are executed by PSR from a pipeline. A step implementer does two things:

1. Integrates with a given product or solution
2. Produces output in a standard format that can be used or validated by other pipeline steps

Available PSR steps can be found here.

An end-to-end example pipeline that uses PSR is documented here.

Documentation

• End-to-End Example Pipeline
• Contributing
• Ploigos Overview
• Python Package Documentation

Install

Latest Release

pip install ploigos-step-runner

Latest Development Release

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple ploigos-step-runner

FAQ

What is a Workflow Runner?

A workflow runner is a Continuous Integration (CI) tool that executes pipelines. Common workflow runners are Jenkins, Tekton, GitHub Actions, and GitLab CI.

What if I want to use a tool that PSR doesn't have an step implementer for?

Each PSR step has a corresponding step_implementer Python module under src/ploigos_step_runner/step_implementers. If you need to use a tool that PSR does not have a step implementer for, you can create one or open a GitHub Issue to request an enhancement. See CONTRIBUTING.md for instructions on creating step implementers.

An AdHoc step implementer is proposed in #131 that would allow for a command or script to be specified in psr.yaml.

Contributing

See CONTRIBUTING.md

ploigos-step-runner's Issues

add flag to do clean or not for any Maven step implementer

purpose

all the step implementer that use 'mvn' have a hard coded 'clean'. There should be a way to disable the 'clean' when needed for those steps

deploy (argocd) - if git-username and git-password provided use them for the clone

issue

if the repo specified for deployment-config-repo required auth for reading and not using SSH keys then will not be able to clone repo because git-username and git-password are not being passed to the git clone operation, only the git push operation.

fix

if git-username and git-password are provided pass them to the git clone operation for the config repository

Stretch - Provide a way to redact sensitive values that are not encrypted using SOPS

Success Criteria:

• Values stored in vault do not show up in the GitHub log output

Initial Tasks:

• Understand the current implementation - https://github.com/ploigos/ploigos-step-runner/blob/main/src/ploigos_step_runner/config/config_value_decryptor.py
• Decide on how to tell PSR or have it detect that something should be redacted (see below for some suggestions, also Aaron thought of telling it everything in a given file is secret)
• Make sure you can run the psr command iteratively for testing (i.e. no long dev cycle that uses a 10 min pipeline run. run it locally or something)
• Plan out how the code will change (classes/high level methods/etc.)
• Write some unit tests
• Actually write the code

Documentation: monorepo support

Creating a separate issue from #99 to deal with monorepo (vs. one build resulting in multiple artifacts).

Assume a git repo with the following layout:

monorepo_base/
    - .git/
    - app1/
    - app2/
    - ...
    - appN/

PSR should provide the ability for a user to specify a subdirectory from the repository root (e.g, app1/) to begin the build from.

Replace 'assert' with 'raise' for Maven logic

For consistency, as well as the ability to control the Exception thrown which validations fail within our logic, replace 'assert' statements with 'raise' statements, e.g.,

assert os.path.exists(pom_file_path), \
    f'Given maven pom file (pom-file) does not exist: {pom_file_path}'

becomes

if not os.path.exists(pom_file_path):
    raise StepRunnerException(
        f'Given maven pom file (pom-file) does not exist: {pom_file_path}'
    )

Please include preferred pronunciation at top of README

My greek is not strong, but it may help to be consistent. I don't think I can manage the official greek pronunciation ... https://translate.google.com/?sl=auto&tl=en&text=ploigos&op=translate

Support Python 3.10

PSR currently won't build using python 3.10. Looks like the toolchain maybe changed with 3.10 no longer supporting the old toolchain.

Success Criteria:

• Contributors can build and run psr using python 3.10.
• The GitHub actions pipelines that check the code include checks for python 3.10.

Is the config docs published anywhere?

Each step has docs:

• https://github.com/ploigos/ploigos-step-runner/blob/main/src/ploigos_step_runner/step_implementers/create_container_image/buildah.py#L1-L31

But are those docs published anywhere?

Externalize Strings

1. Research string externalization libs for python
2. Pick a library
3. Find all human-readable strings in step runner
4. Externalize these human-readable strings using the selected python library

disable-maven-central-repository

https://github.com/rhtconsulting/tssc-python-package/blob/3d41440a44c161839bba23bcdaad950013bc5a25/tssc/utils/maven.py#L36

https://stackoverflow.com/questions/12789488/how-do-you-stop-maven-from-trying-to-access-http-repo-maven-apache-org

https://stackoverflow.com/questions/4997219/disable-maven-central-repository

stop this from happening:

[�[1;34mINFO�[m] 
[�[1;34mINFO�[m] -------------------------------------------------------
[�[1;34mINFO�[m]  T E S T S
[�[1;34mINFO�[m] -------------------------------------------------------
[�[1;34mINFO�[m] Running org.acme.rest.json.�[1mFruitResourceTest�[m
Downloading from central: https://repo.maven.apache.org/maven2/io/cucumber/messages/maven-metadata.xml
Downloading from internal-mirror: https://nexus-devsecops.apps.spartawest.rht-set.com/repository/maven-public/io/quarkus/quarkus-resteasy-jsonb-deployment/1.4.2.Final/quarkus-resteasy-jsonb-deployment-1.4.2.Final.pom

Create a Developer Contributor Guide

need way to import trusted CAs

purpose

when orgs are using certificates for their services signed by an internal CA need a way to importat that CA as trusted for use by all commands (import into truststore) rather then forcing people to disable TLS verification

possible implementation

Update reference workflows (currently Jenkins/Tekton) to inject the cluster custum pki into the workflow containers:

https://docs.openshift.com/container-platform/4.6/networking/configuring-a-custom-pki.html#certificate-injection-using-operators_configuring-a-custom-pki

dont include substep in the generated evidence

issue

currently the Sub Step name is being included in the evidance, but that means OPA rules would have to have the sub step name in their paths (unless we can do wild card paths), but we should just get rid of the sub step name in the json evideance to simplify things

background

ploigos-step-runner/src/ploigos_step_runner/step_implementers/generate_evidence/generate_evidence.py

Line 209 in 8bae426

sub_step_name = previous_step_result.sub_step_name

RFE: Add function to StepImplementer abstract class that returns list of config values that should be considred "private"/"secret"/"protected"

Issue

if someone passes a value typically considered "secret" to the PSR via non encyrpted means, such as a a flag to the PSR cli or iva a config file that wasn't encrypted, PSR has no way of knowing that value should be protected and will print it to stdout without any obfuscation.

while the defualt stance of the PSR is that any "secret" values should always be encrypted at rest, there is a case to be made for having another level of protect htat for certain config values they are always treated as "protected" and their values always obfuscated whether user passed them in as encrypted or not.

Possible solution

Update StepImplemetner abstract class and its implementations so that as they read in config values that any value considered "always secret" gets registered with the DecryptorUtils as something to obfuscate.

Something to watch out for is currently the StepImplmeenters do not know which values are encrypted by users and which arnt' because that is handled independetly and invisibly to the StepImplementers so have to deal with case of value being pased in encrypted for a "protected config value" and then the StepImplemtner registering that vlaue again with the DecryptorUtils, maybe just having some de-dup logic in the DecryptorUtils or something.

Origional Post

Configuration entries are printed to stdout during a pipeline run. This results in passwords and tokens being visible in cleartext, like so:

Global Configuration Defaults
    {'application-name': 'ref-quarkus-mvn-jenkins-std',
     'container-registries': {'quay-devsecops.apps.ocp4.adamgoossens.com': {'password': 'xxxxxxxxxxxxx',
                                                                            'username': 'tssc'},
                              'registry.redhat.io': {'password': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
                                                     'username': 'xxxxxxxxxxxxxxx|xxxxxxxxxxxxxxxxxx'}},

Either we should avoid printing any content that is marked as sensitive, or ensure everything is encrypted at rest in the configuration tree so no cleartext passwords/tokens will appear in stdout.

Implement Step Runner for Performance Testing

RFE: tag-source/git step implementer needs ssh key parameter

purpose

currently no way to use an ssh key to do git tagging (unless project happened to be checked out with an ssh key with write permissions).

suggested fix

1. add a utility function to import arbitrary ssh keys (like we have for pgp keys)
2. update tag-source/git to take an ssh key paramter and import it using the ultity function
3. any other updates to tag-source/git to use the ssh key

push-artifacts (maven) - support skip tls verification

purpose

need a way to tell maven to skip tls verification

suggestion

add a skip-tls or similar flag.

when set to True do some form of

-Dmaven.wagon.http.ssl.insecure=true 
-Dmaven.wagon.http.ssl.allowall=true 
-Dmaven.wagon.http.ssl.ignore.validity.dates=true

RFE: document building custom Decryptors for external secret management systems

Issue

Currently, PSR uses SOPS to store encrypted values and uses a SOPS decryptor to extract that value on access to the config item.

Many organisations will have third party systems for secret storage (e.g. Vault, Conjur), and these might be the 'official' secret store in those organisations. It would be ideal to be able to use PSR to retrieve and use secrets from these systems, referencing them from config like so (for example): my_vault_secret: VAULT_ENC[/path/to/my/secret]

Possible Solution

This is possible through the ConfigValueDecryptor classes, however it needs to be documented. The documentation would need to cover:

• The base class and required methods
• The flow from encrypted value to decrypted output.
• Use of the obfuscation list to prevent sensitive data from being dumped to stdout/stderr
• How to store and use 'bootstrap secrets' for the Decryptor, e.g. the authentication credentials for a secret store's API.

With documentation in place it becomes easier for additional contributions for other secret store mechanisms.

maven-no-transfer-progress is not being used when updating pom version - breaking workflows with untrusted maven repo

issue

when calilng run_maven in the MavenDeploy class to update the version the maven-no-transfer-progress config value is not being checked or added to the flags.

backround

It was assumed that updating the verison was a local only operation and it wouldn't be a problem. turns out it is

fix

need to copy or abstract the logic form MavenGeniric::_run_maven_step to the MavenDeploy when doing the version update

sonarqube java keystore must be set via env and not command line flag

issue

ced0ac5 was an attempt to set the javakeystore sonaqube uses, but turns out sonarqube command can't receive JVM flags directly.

fix

update the sonarqbute step implementer to pass SONAR_SCANNER_OPTS via the _env param to the sh command rather then passing a -D flag to the mvn command.

see

• https://amoffat.github.io/sh/sections/special_arguments.html#env
• https://github.com/ploigos/ploigos-step-runner/blob/main/src/ploigos_step_runner/step_implementers/static_code_analysis/sonarqube.py#L216-L227

Automate the deletion of preview environments

As a software developer, when I close a pull request I need the preview environment managed by ArgoCD to be deleted.

Service mesh support for UAT step

Realized I opened this originally in the wrong repo

We are currently using the UAT step to perform automated testing via a project deployed via ArgoCD. We have recently switched from using route objects to using service mesh (specifically ingress-gateway) for exposing our endpoints. However the UAT step only parses though ingress and route objects returned by deployment. Since these are not being deployed by our application the UAT step is failing because it gets 0 host urls. We would like to see if there is any current planned support for service mesh or if a PR for service mesh support being added to UAT would be a welcome addition.

Default value for the -c argument

Currently, the -c flag for psr is required. We support a default value so users running it manually do not have to type it every time. The default value should be "psr.yaml".

Success Criteria;
Running psr -s generate-metadata is equivalent to running psr -c psr.yaml -s generate-metadata.

Deploy URl hardcoded to RHT-Set

Make the deployment urls for the hooks dynamically generated for the app

LOW PRIORITY - effective-pom.xml has http: urls for xsds

If an application runs some kind of coding style scan that looks for http: urls (instead of https:), that scan turns up a finding because PSR generates an effective-pom.xml file that has xsd imports using the http protocol.

The maintainer of the app being built by ploigos can work around this by suppressing the finding, but that is not ideal because it (slightly) increases the effort to onboard the app to PSR.

We should generate effective-pom.xml using https: urls for the xsd imports.

Example:

• An app running the maven checkstyle plugin turns up warnings like

Error:  [ERROR] step-runner-working/unit-test/effective-pom.xml:[5,12] (extension) NoHttp: http:// URLs are not allowed but got 'http://maven.apache.org/plugins/maven-help-plugin/'. Use https:// instead.

• App developer has to add a suppression rule like this (this is not a great rule but you get the idea).

<suppress files=".+\.(jar|git|ico|p12|gif|jks|jpg|svg|xml)" checks="NoHttp"/>

How to start implementing the fix:
Change this line in the relevant unit test to check for the new behavior
https://github.com/ploigos/ploigos-step-runner/blob/main/tests/utils/test_xml.py#L36

deploy/argocd - underscores in branch name cause error

The ArgoCD step implementer fails with the following error underscore when the name of the branch contains underscores:

metadata.name: Invalid value: \"nc-covidsafe-ui-feature-intell_shot_dates-dev\": a lowercase RFC 1123 

re-enable bandit testing

purpose

apparently our CI bandit call was never actually scanning due to a missing -r flag. need to re-enable and then fix all the errors

Support DC OCTO team using PSR to deploy a workload

This will involve updating an app to use the minimal Jenkins pipeline - https://github.com/ploigos/ploigos-jenkins-library/blob/main/vars/ploigosWorkflowMinimal.groovy

David Ford & Chris May will be doing the billable work to get this working. The rest of the TSSC team will support as needed.

Integration Test infrastructure missing

This code is no longer working due to the cluster no longer existing

.github\workflows\pull-request.yml

integration-test:
runs-on: ubuntu-latest
concurrency:
group: ${{ github.ref }}
cancel-in-progress: true
steps:
- name: Checkout 🛎️
uses: actions/checkout@v2

  - name: Integration Tests
    env:
      OPENSHIFT_URL: ${{ secrets.OPENSHIFT_URL }}
      OPENSHIFT_USER: ${{ secrets.OPENSHIFT_USER }}
      OPENSHIFT_PASSWORD: ${{ secrets.OPENSHIFT_PASSWORD }}
      PSR_BRANCH: ${{ github.ref }}
    run: |
      ./tests/integration/run-integration-tests.sh

deploy - add helm StepImplementer

purpose

for people who dont want to use argo, or want to create Argo CRs using helm rather then API (there are reasons), we need a generic helm step implementer.

requirements

minimum

• take parameter for relative path to helm chart to run local to already checked out git project
• takes parameters with list of arbitrary parameters to pass to helm
• uses the helm secrets plugin so that sops things are decrypted (assumes pgp keys or si8mialr are already installed/configured, but handles error case where they are not)
• uses helm secrets upgrade --install rather then helm secrets install so that it works for an initial install or upgrade of already installed helm chart
• takes optinoal paramter for namespace to install helm chart into, if not given crafts a namespace name like we do in Argo step implementer

stretch or future enhancements

• take parameter for git project to checkout with helm chart to run and then use the relative path to helm chart in that project rather then the primary project
• rather then using local source helm chart be able to configure arbitrary helm chart repository and then run a binary helm chart from that repository

Reporting Step Implementer

1. Design how data will be written into a report output
2. Design an interface for accessing this data

Tests fail when skopeo is not installed on local machine

Two of the 'test_skopeo_push_container_image.py' tests fail when the steprunner tests are run on a machine that does not have skopeo installed.

Failing tests:

• TestStepImplementerSkopeoSourceBase.test_run_step_fail_run_skopeo
• TestStepImplementerSkopeoSourceBase.test_run_step_pass

Output:

============================================================================================= short test summary info =============================================================================================
FAILED tests/step_implementers/push_container_image/test_skopeo_push_container_image.py::TestStepImplementerSkopeoSourceBase::test_run_step_fail_run_skopeo - AttributeError: <module 'sh' from '/home/cthiel/pl...
FAILED tests/step_implementers/push_container_image/test_skopeo_push_container_image.py::TestStepImplementerSkopeoSourceBase::test_run_step_pass - AttributeError: <module 'sh' from '/home/cthiel/ploigos-step-...
==================================================================================== 2 failed, 674 passed, 7 skipped in 9.39s =====================================================================================
ERROR: InvocationError for command /home/cthiel/ploigos-step-runner/.tox/test/bin/python -m pytest --cov --cov-report=term-missing --cov-report=xml (exited with code 1)

Once skopeo is installed on the local machine, these tests run without issue and all tests pass

Expected behavior: All tests should pass regardless of what tools are present on the local machine

Running the uat step twice in a Jenkins pipeline throws an exception

We have a jenkins pipeline that executes the UAT step twice - once for each of two environments. We receive the following error in the jenkins log when the second UAT step is run:

Fatal error calling step (uat): Can not add duplicate StepResult for step (uat), sub step
 (ploigos_step_runner.step_implementers.shared.MavenGeneric), and environment (DEV).

When viewing the corresponding code, the commentary defines a process for rectifying the duplicate step results however the actual code throws an exception.

Ideally, this use case would be supported otherwise documentation should be updated to indicate otherwise.

cc @HunterGerlach

PSR Step - "Break Glass Option"

An implementer which can run a bash script or single command.

Rationale: Sometimes when you're prototyping you just want to run that CLI command instead of implementing a step implementer right away.

Success Criteria:

When I put something like this in psr.yaml:

  uat:
  - implementer: ploigos_step_runner.step_implementers.shared.ad_hoc.AdHoc
    config:
      command: run-acceptance-tests.sh

And I run

psr -c psr.yaml -s uat

Then PSR should execute run-acceptance-tests.sh

Notes:

• To do this, create a new step implementer in the shared folder here that we can reference from any step - https://github.com/ploigos/ploigos-step-runner/tree/main/src/ploigos_step_runner/step_implementers/shared
• We should open a follow-on story to give the user a way to register results by writing values to files in the script

make the OPA policy query path a parameter

problem

currently the path to the policy query path is hard coded as a contant, it shoudl be a configuraiton value with a default.

background

ploigos-step-runner/src/ploigos_step_runner/step_implementers/audit_attestation/opa.py

Line 45 in 8bae426

DEFAULT_WORKFLOW_POLICY_QUERY = "data.workflowResult.passAll"

deploy/argocd - need option to enable Prune Resources

purpose

currently no way to enable the argocd Prune Resources option via the ArgoCD StepImplementer.

new/update step implementer - create-container-image - buildah native vs buildah bud

purpose

currently the buildah StepImplementer for the create-container-image step uses buildah bud which runs against container files. We should re-name that step implementer as buildah_bud and create a new one that is buildah that uses the native buildah build commands rather then the old fassioned container file.

reason

if using native buildah commands then we could start to do fun things like have a (sub) step implmenter in the workflow that does atuomatic oscap remediation. though this may be possible already by mounting the image built with buildah bud and then running the oscap remediation using buildah commands, then re-sealing the image before ever pushing....

distutils is deprecated

issue

we use distutils for the strtobool function, but it seems the module is being depreciated in newer versions of python. quick googling around and all i see is others bringing the funciton into their own repos there seems to be no replacment in community.

fix

bring the stringtobool funcitonality into our util funcitons

Rename "application-name" and "service-name"

Purpose

I have never liked the "application" and "service" monikers, but could never think of a better option. After conversation with @BillBensing and @raffaelespazzoli a new idea came out.

a system are composed of components. Where system replaces applicaiton and component replaces service. I really like this idea becasue we already use the component lanugae when talking about the infrastrcture that supports everything so it fits well here. I think it also avoids the complication of people having a defintion in their head what an app is and what a service is.

Proposal

Since both of these values are fundemental to a lot of steps I wnat to add a @property to StepImpleneter base class to get them that looks like:

    @property
    def system_name(self):
        return self.get_value(['system-name', 'application-name'])

    @property
    def component_name(self):
        return self.get_value(['component-name', 'service-name'])

And then find and replace all instances of self.get_value('applicaiton-name') and self.get_value('service-name') with a call to that property that now lives in the base StepImplementer.

This means that we are botgh backwards compatabile, and should we ever choose to fullly depcricate the old property value, we only have to do it in one place.

Decisions

what should the new proprties be called, i get in nots between system or system-name and component or component-name i like the -name for explicity and dont like it for being overly verbose.

Or maybe everyone hates the idea of changing the names, but maybe everyone can at least buy off on moving to these to properties in the base StepImpleneter class?

Investigate use of kube-linter and need for new step or not

Purpose

Investigate https://github.com/stackrox/kube-linter and see if it fits in as another step implementer for the validate-environment-configuration step or if it falls into some new category.

Other musings

Currently we run validate-environment-configuration post deployment and use https://github.com/stelligent/config-lint. By guess is that both config-lint and kube-linter can be used post deployment by doing what we do now which is download the namespaces kube state. Buuuuut...... this kube-linter brings to mind that there is probably a reason to add a new step to validate the deployment config BEFORE deployment as well. The reason we are doing it after currently was the specific use case of looking for the runtime injection of a service mesh which can't be found in the static config because is injected by overlay operators at runtime, which means we probalby need both validate-environment-configuration and a new deployment-config-static-code-analysis step or something similar.

package maven step implementer should support pom files that build multiple artifacts

current

step_implementers.package.maven.Maven only supports pom.xml files that build a single resource

future

support pom files that build mutliple artifacts

need ability to sign artifacts from authed sources

issue

currently the rekor signing step can only sign artifacts pulled from sources that do not require authentication.

fix

need ability to pass credentials for artifact store that artifacts are being pulled from to sign.

see

• https://github.com/ploigos/ploigos-step-runner/blob/main/src/ploigos_step_runner/step_implementers/shared/rekor_sign_generic.py#L12-L15
  • need artifact-to-sign-source-username and artifact-to-sign-source-password otpional parameters
• https://github.com/ploigos/ploigos-step-runner/blob/main/src/ploigos_step_runner/step_implementers/shared/rekor_sign_generic.py#L245-L247

• this needs ability to pass username/pass

• https://github.com/ploigos/ploigos-step-runner/blob/main/src/ploigos_step_runner/utils/file.py#L63-L66
  • needs to accept optional username/pass params

Implement step runners for JavaScript

1. Create a step runner to JavaScript build
2. Create a step runner for JavaScript testing

Implement step runners for Python

1. Create a step runner to python build
2. Create a step runner for python testing

deploy/argocd - need option to enable Self Heal

purpose

currently no way to enable the argocd Self Heal option via the ArgoCD StepImplementer.

Sonar Scanner Token

The sonar scanner container does not have the latest code with the token login.

Parallel step processing results in missing step artifacts/evidence later in the pipeline

When two steps run in parallel (e.g. the compliance and vulnerability scans in the Jenkins-based pipeline) the results from one or more of the steps will be missing from the resulting artifacts that are bundled up.

I haven't tested this with more than two steps in parallel, but I suspect the result will be only one of the steps is retained.

Here's the sequence of events as I understand it:

1. Step A kicks off, loads the workflow history from the Pickle file (here).
2. At the same time, Step B runs in parallel and also loads the workflow history from the Pickle file.
3. Neither of these two is aware of the other.
4. Step A completes, dumps its results to the Pickle and YML files (here).
5. Step B completes, dumps its results to the Pickle and YML files and completely overwrites what Step A added.
6. Step A's artifacts and evidence is now missing from the rest of the pipeline.

I've got a snapshot of the working directory during the run, post the compliance/vulnerability checks, that illustrates the issue. Happy to make it available if it's useful, just need to find somewhere to upload it.

Aggregate Unit Test Results for Multi-module Projects

As a developer working on a multi-module maven project
I want unit test results aggregated from all sub modules
So that I have unit test pass / fail /error evidence aggregated in a single location

The current workaround is to use the additional-artifacts configuration to capture multiple unit test XML files (see below). This does not aggregate all the final results but does capture the details:

  unit-test:
  - implementer: MavenTest
    config:
      test-reports-dir: service/target/surefire-reports
      additional-artifacts:
      - service/target/surefire-reports
      - kjar/target/surefire-reports
      - model/target/surefire-reports

There should be a way to aggregate the results, for example by specifying an array of test-reports-dir