open-toolchain / tekton-catalog

Catalog of reusable Tasks usable in Continuous Delivery Tekton Pipelines

Open-Toolchain Tekton Catalog

Catalog of Tekton Tasks usable in Continuous Delivery Tekton Pipelines

Notes:

• These tasks are usable with Continuous Delivery Tekton Pipeline Worker Agent (Tekton definition with apiVersion: v1beta1). These tasks have been updated following migration path described in https://github.com/tektoncd/pipeline/blob/v0.11.2/docs/migrating-v1alpha1-to-v1beta1.md
• If you want v1alpha1 resources, you need to reference the tekton_pipeline0.10.1 tag (or tekton_pipeline0.10.1_workspace tag to have v1alpha1 resources using workspaces).
• When moving from from tag tekton_pipeline0.10.1, tekton_pipeline0.10.1 and/or branch tkn_v1beta1 to use masterbranch of this catalog, take a look at breaking changes section

Tasks

Cloud Foundry related tasks

• cf-deploy-app: This task allows to perform a deployment of a Cloud Foundry application using ibmcloud cf commands.

IBM Cloud Container Registry related tasks

• icr-containerize: This task is building and pushing an image to IBM Cloud Container Registry. This task is relying on Buildkit to perform the build of the image.

• icr-execute-in-dind: This task runs docker commands (build, inspect...) that communicate with a sidecar dind, and push the resulting image to the IBM Cloud Container Registry.

• icr-execute-in-dind-cluster: This task runs docker commands (build, inspect...) that communicate with a docker dind instance hosted in a kubernetes cluster (eventually deploying the Docker DinD if needed), and pushes the resulting image to the IBM Cloud Container Registry.

• icr-check-va-scan: This task is verifying that a Vulnerability Advisor scan has been made for the image and process the outcome of the scan.

• icr-cr-build - deprecated: The ibmcloud cr build command is deprecated. If you use the icr-cr-build Tekton task, you can migrate to one of the three above Tekton tasks to build container images. For more information about this replacement, see the IBM Cloud™ Container Registry is Deprecating Container Builds blog post.

IBM Cloud Code Risk Analyzer scanners related tasks

• cra-discovery: This task accesses various source artifacts from the repository and performs deep discovery to identify all dependencies (including transitive dependencies).
• cra-bom: This task creates a Bill-of-Material (BoM) for a given repository that captures pedigree of all the dependencies and is collected at different granularities.
• cra-cis-check: This task runs configuration checks on kubernetes deployment manifests.
• cra-vulnerability-remediation: This task finds out vulnerabilities for all application package dependencies, container base images and os packages.
• cra-comm-editor: This task creates comments on Pull Requests and opens issues regarding bill of material and discovered vunerabilities.
• cra-terraform-scan: ## This task scans ibm-terraform-provider files for compliance issues.

IBM Cloud Devops Insights related tasks

• doi-publish-buildrecord: This task publishes build record to DevOps Insights
• doi-publish-testrecord: This task publishes test record(s) to DevOps Insights
• doi-publish-deployrecord: This task publishes deploy record to DevOps Insights
• doi-evaluate-gate: This task evaluates DevOps Insights gate policy

Git related tasks

• git-clone-repo: This task fetches the credentials needed to perform a git clone of a repo specified by a Continuous Delivery toolchain and then uses them to clone the repo.
• git-set-commit-status: This task is setting a git commit status for a given git commit (revision) in a git repository repository integrated in a Continuous Delivery toolchain.

IBM Cloud Kubernetes Service related tasks

• iks-fetch-config: This task is fetching the configuration of a IBM Cloud Kubernetes Service cluster that is required to perform kubectl commands.
• iks-contextual-execution: This task is executing bash snippet/script in the context of a Kubernetes cluster configuration.
• iks-deploy-to-kubernetes: This task allows to perform scripts typically doing deployment of a Kubernetes application with ibmcloud ks cli and kubectl cli configured for a given cluster.

Linter related tasks

• linter-docker-lint: This task performs a lint on the given Dockerfile using Hadolint.

Signing - Docker Content Trust related tasks

• signing-dct-init: This task initialize Docker Content Trust GUN/repository
• signing-dct-sign: This task performs a Docker Content Trust signature on a given image
• signing-dct-enforcement-policy: This task installs Container Image Security Enforcement on a given cluster.

Slack related tasks

• slack-post-message: This task posts a message to the Slack channel(s) integrated to your Continuous Delivery toolchain.

SonarQube related tasks

• sonarqube-run-scan: This task starts a SonarQube scan for the code in a workspace using the SonarQube server integrated to your Continuous Delivery toolchain.

Tester related tasks

• tester-run-tests: This task allows to invoke a script to execute test.

Open-Toolchain related tasks

• toolchain-build: This task performs build operation(s) on the given workspace. Default build operations managed are maven build for instance.
• toolchain-extract-value: This task extracts values from the desired config map with a given jq expression.
• toolchain-publish-deployable-mapping: This task creates or updates a toolchain deployable mapping for a Continuous Delivery toolchain.

Breaking Changes

when moving from tag "tekton_pipeline0.10.1"

• These tasks are using kebab-case style for EVERY parameters names. So parameter pathToContext (in previous versions of the tasks) has been renamed as path-to-context, parameter clusterName has been renamed to cluster-name and so on...

• communication folder has been renamed to slack folder

• Some tasks has been renamed to match the following name format <category alias>-<task> where category alias is depending on the folder containing the tasks:

  Folder/Category	Category alias
  cloudfoundry	cf
  container-registry	icr
  devops-insights	doi
  git	git
  kubernetes-service	iks
  slack	slack
  toolchain	toolchain

  The task new names are listed in the following table:

  Folder	Old task name	New task name
  container-registry	containerize-task	icr-containerize
  container-registry	cr-build-task	icr-cr-build
  container-registry	execute-in-dind-task	icr-execute-in-dind
  container-registry	execute-in-dind-cluster-task	icr-execute-in-dind-cluster
  container-registry	vulnerability-advisor-task	icr-check-va-scan
  git	clone-repo-task	git-clone-repo
  git	set-commit-status	git-set-commit-status
  kubernetes-service	fetch-iks-cluster-config	iks-fetch-config
  kubernetes-service	kubernetes-contextual-execution	iks-contextual-execution
  slack	post-slack	slack-post-message

• Tasks that use workspace(s) may have changed the expected workspace name. Here is the list of the breaking changes for the expected workspace name

  Folder	Task	Old workspace name	New workspace name	Description
  container-registry	icr-containerize	workspace	source	A workspace containing the source (Dockerfile, Docker context) to create the image
  container-registry	icr-cr-build	workspace	source	A workspace containing the source (Dockerfile, Docker context) to create the image
  container-registry	icr-execute-in-dind	workspace	source	A workspace containing the source (Dockerfile, Docker context) to create the image
  container-registry	icr-execute-in-dind-cluster	workspace	source	A workspace containing the source (Dockerfile, Docker context) to create the image
  container-registry	icr-check-va-scan	workspace	artifacts	Workspace that may contain image information and will have the va report from the VA scan after this task exection
  git	git-clone-repo	workspace	output	Workspace where the git repository will be cloned into
  git	git-set-commit-status	workspace	artifacts	Workspace that may contain git repository information (ie build.properties). Should be marked as optional when Tekton will permit it
  kubernetes-service	iks-fetch-config	workspace	cluster-configuration	A workspace where the kubernetes cluster config is exported
  kubernetes-service	iks-contextual-execution	workspace	cluster-configuration	A workspace that contain the kubectl cluster config to be used

when moving from tag "tekton_pipeline0.10.1" and/or branch "tkn_v1beta1"

• Tasks that are expecting a secret to retrieve apikey and/or secret values have been updated to use the default secret secure-properties injected by Continuous Delivery Tekton Pipeline support. The updated tasks are:

  • icr-check-va-scan
  • icr-containerize
  • icr-cr-build
  • icr-execute-in-dind
  • icr-execute-in-dind-cluster
  • git-clone-repo
  • git-set-commit-status
  • iks-fetch-config

  Note: As a reminder, in previous version (before secure-properties injection by CD tekton support), the default was set to cd-secret

Criteria for Code Submission

To ensure code quality, protected branches will be enabled soon, and every PR that is to be merged to master will run CI tasks for code quality. These could (and should) be set up for local development environments as well.

Code quality checks currently enabled:

• yaml lint - using yamllint-rules.yaml as configuration file: yamllint --config-file yamllint-rules.yaml .
• tekton task lint: tekton-lint '**/*.yaml'
• Tasks definition validation: check_tasks.sh

tekton-catalog's Issues

Git clones in response to github tag creation fails

Scenario:
Creating a release/tag in github, results in a git reference of refs/tags/v0.0.2

Issue:
With the git clone task, the references are checked for the format refs/heads/xxx but not tags. Therefore any tagged release fails.

Will create a PR :-)

Fix README in container-registry

https://github.com/open-toolchain/tekton-catalog/blob/master/container-registry/sample/README.md has these duplicate lines:

It also contains a buildkit-no-resources EventListener definition which is the providing the same example but without the needs to define PipelineResources for image as it uses the task's parameter image-url to provide the information.

It also contains a buildkit-no-resources EventListener definition which is the providing the same example but without the needs to define PipelineResources for image as it uses the task's parameter image-url to provide the information.

Most likely you wanted to document buildkit-no-resources and buildkit-no-image-url, but the latter is missing.

Add README to samples under kubernetes-service

https://github.com/open-toolchain/tekton-catalog/tree/master/kubernetes-service/sample
is missing a README and a sample for the iks-deploy-to-kubernetes

Add documentation or link on how to use the catalog

The README does not contain any instructions on how to (re)use the assets in this catalog. It would be great to have at least some links to outside documentation on how to install and use an asset.

The linked Continuous Delivery service and its Tekton section do not have any discussion of this catalog.

kubernetes-service: "ibmcloud ks cluster config" with "--export" parameter?

the tasks defined under kubernetes-service use the command ibmcloud ks cluster config and a parameter --export. That parameter is not documented here: https://cloud.ibm.com/docs/containers?topic=containers-cli-plugin-kubernetes-service-cli#cs_cluster_config

When trying that command locally on version 1.x, that parameter causes an error. Is it still valid?

Allow set-status pending when already pending to be a no-op

Setting status to pending in gitlab when status is already pending causes a HTTP 400
Can we check the status if we are in gitlab AND we want to set to pending - and if already pending, then do not attempt to set the status ?
@jauninb FYI

Containerize task fails using IBM Managed Shared Workers (public worker)

When using containerize task in a pipeline to create docker image - https://github.com/open-toolchain/tekton-catalog/blob/master/container-registry/task-containerize.yaml - , recently we're receiving this error:

error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to read dockerfile: failed to mount /home/user/.local/tmp/buildkit-mount136740543: [{Type:bind Source:/tekton/home/.local/share/buildkit/runc-native/snapshots/snapshots/1 Options:[rbind ro]}]: operation not permitte

task-deploy-to-kubernetes fails due to not being authenticated to OpenShift cluster

I followed the tutorial https://www.ibm.com/cloud/architecture/tutorials/develop-kubernetes-app-with-code-risk-analyzer, but pipeline fails on deploy-to-kubernetes and check-health steps due to not being authenticated to cluster, with the following messages:

Check ability to get a kubernetes deployment in test-proj using kubectl CLI
error: EOF

After I added some additional testing commands I noticed that user is not authenticated to cluster:

++ echo 'Check ability to create a kubernetes deployment in test-proj using kubectl CLI #### TESTING'
++ oc projects
Check ability to create a kubernetes deployment in test-proj using kubectl CLI #### TESTING
Error from server (Forbidden): namespaces is forbidden: User "system:anonymous" cannot list resource "namespaces" in API group "" at the cluster scope

I had to add scripts from commons repo to my app and change them to include the following code that authenticates to OCP, which I took from your other tutorial:

ibmcloud ks cluster config -c "${PIPELINE_KUBERNETES_CLUSTER_NAME}";
if which oc > /dev/null && ibmcloud ks cluster get -c "${PIPELINE_KUBERNETES_CLUSTER_NAME}" --json | jq -e '.type=="openshift"' > /dev/null; then oc login -u apikey -p "${PIPELINE_BLUEMIX_API_KEY}" ; fi;

So looks like task definition is missing proper authentication to cluster that it is deploying to....

[suggestion] Run tekton-lint on commits and PRs

May I suggestion running tekton-lint on every commit and PR?

If could also be used for branch protection.

You can use this to run it via docker:

# using docker
docker run \
    --rm \
    --tty \
    "--env=TERM=${TERM}" \
    "--volume=${PWD}:/src:ro" \
    "--workdir=/src" \
    node \
    npx tekton-lint --format=stylish "**/*.yaml"

or if you have node installed:

# using node 12+
npx tekton-lint --format=stylish "**/*.yaml"

I can submit a PR for a script/lint and/or for the CI system of your choice.

git-set-commit-status task fails when pipeline triggered from non-SCM trigger

Conditional branching of paths is hard to achieve in Tekton, so I propose that this task looks at the trigger of the pipeline and simply returns successfully if executed within the context of the pipeline being triggered by something like a timed or manual trigger.

Problem with ' in the additional-tags-script parameter value for icr-containerize

additional-tags-script parameter for https://github.com/open-toolchain/tekton-catalog/tree/master/container-registry#icr-containerize is not took in account/working well when there is some ' in the script/parameter value

Add support for Schematics

It would be useful to have tasks to drive a schematics workspace (new/plan/apply/destroy/output support)

I found some in https://github.com/open-toolchain/schematics-toolchain/tree/master/.tekton but moving them to the catalog would be a better option