jenkins-x / jx3-pipeline-catalog

the default tekton catalog for Jenkins X 3.x

jx3-pipeline-catalog

The default pipeline catalog for Jenkins X 3.x

Jenkins X 3.x comes with its own default pipeline catalog for different languages, tools and frameworks. This catalog contains reusable steps, Tasks, Pipelines and Packs you can use on any project.

For more information check out the Jenkins X 3.x support for Tekton Catalog

Contents

• tasks a reusable folder of tasks and associated triggers
• packs contains the language and/or framework specific packs containing tekton pipelines and associated files used by the pipelines such as Dockerfile or helm charts.
  • e.g. the javascript pack has the Jenkins X pipelines at packs/javascript/.lighthouse/jenkins-x
• helm contains reusable helm charts that are imported into the various folders in packs such as packs/javascript/charts to share charts across the different programming languages

Custom Pipeline Catalogs

(Blog article)

To use your own custom pipeline catalog, you can fork this catalog to make changes for your team or share between teams in your company. You can make as many catalogs as you like and put whichever catalogs you want in the extensions/pipeline-catalog.yaml file of your cluster git repository of your Jenkins X 3.x install, like for example:

# Source: <boot-repo>/extensions/pipeline-catalog.yaml
apiVersion: project.jenkins-x.io/v1alpha1
kind: PipelineCatalog
spec:
  repositories:
  - label: Your Pipeline Catalog
    gitUrl: https://github.com/<your-org>/jx3-pipeline-catalog
    gitRef: master

For more detail there's the configuration reference here.

Then when developers create a new quickstart or import a repository developers will be asked to pick the catalog they want from your list if there is more than one, or the configured catalog is silently used.

This gives you complete freedom to configure things at a global, team or repository level while also making it easy to share changes across projects, teams and companies.

Upgrading Charts

Many of these packs contain helm charts such as packs/javascript/charts which are managed via kpt.

If the helm/charts folder is modified you can upgrade the charts in each pack via:

jx gitops upgrade --ignore-yaml-error

jx3-pipeline-catalog's Issues

Javascript quickstarts fail due to git related tasks

I tried to use the react and node-http quickstarts. Both of them failed to create their pipeline. jx pipeline lint reveals:

WARNING: ignoring missing user name in git credentials file: /Users/brad/.git-credentials URL: https://:@github.com
FILE                                   STATUS
.lighthouse/jenkins-x/triggers.yaml    OK
.lighthouse/jenkins-x/pullrequest.yaml failed to unmarshal YAML file .lighthouse/jenkins-x/pullrequest.yaml: failed to validate generated PipelineRun: invalid value: git-clone: spec.pipelinespec.tasks[0].taskSpec.steps[2].name
invalid value: git-merge: spec.pipelinespec.tasks[0].taskSpec.steps[3].name
.lighthouse/jenkins-x/release.yaml     failed to unmarshal YAML file .lighthouse/jenkins-x/release.yaml: failed to validate generated PipelineRun: invalid value: git-clone: spec.pipelinespec.tasks[0].taskSpec.steps[1].name

I am not sure if it is related but my cluster has a clusterTask called git-clone. Removing that git-clone clusterTask does not seem to resolve the problem.

If I remove metadata.annotations.lighthouse.jenkins-x.io/prependStepsURL: https://raw.githubusercontent.com/jenkins-x/jx3-pipeline-catalog/9ef3f3c107abc35b8f49d2e17d3af9ba5c5ef932/tasks/git-clone/git-clone-pr.yaml it sems to resolve the lint issue. I figured this out because I imported a javascript project that didn't have this problem and a diff of their pullrequest.yaml was the only difference. The working copy did not have the offending annotation.

I don't think it is related but I will mention it here: after fixing this linting problem, committing the changes and pushing them to the repo. pipelineRuns are created which in turn create taskRuns but the taskRuns never start and do not get a status block.

Automatically update pipeline steps when versionstream is updated

Pipeline files should receive automatic updates when a newer version of jx is released.

It's important for us to keep to the same major version.
Unsure about minor but would certainly work for patch.

Ie 2.1.7 -> 2.1.8
3.0.1 -> 3.0.3

This will save a lot of headaches when updating microservices pipelines.

PHP example not working out-of-the-box

Hello,

The php example is not working OOTB because the chart tries to connect to port 8080 while the webserver listen on the port 80:

jx3-pipeline-catalog/packs/php/charts/values.yaml

Line 51 in 8302205

internalPort: 8080

This was fixed in jenkins-x-buildpacks/jenkins-x-kubernetes@7ebf0f6 of the build-packs, but apparently the changes are not automatically merged in this repository?

Provide tasks for use in a git flow branching strategy

The branch model default in Jenkins-X is trunk based. This is probably fine for most project, but sometimes gitflow is needed. I suggest reusable tasks for implementing gitflow based pipelines are created.

Add Makefile,Kptfile and .helmignore in .helmignore

It's a minor thing but I think those files should not be included in helm chart by default. Or maybe it's just me that have no use case for them :).

./README.md
./Chart.yaml
./Kptfile
./Makefile
./templates
./templates/release.yaml
......................................
./templates/hpa.yaml
./.helmignore
./values.yaml

Maven tasks should have mvn-install for the release pipeline

At present, the maven tasks use "mvn deploy" in the release pipeline. This pushes packages to the maven registry, which may be useful in some situations, but in many situations, these packages will never be used and simply take up space. We could add an "mvn install" step, which does the same, but without pushing packages. Then each user could modify the pipeline in the builds and select if they want to use "mvn install" or "mvn deploy".

Release pipeline git-merge step merges master to base_branch(on which release is made)

On release pipeline currently git-merge step merges master to base_branch(on which release is made, eg staging), whenever release is made on such non master base branches.
This shouldn't not be hard func to merge master to every release branch present in a repo whenever merge is made to them.

lets add CI="true" env var on every stepTemplate in the pipeline library

to avoid issues with git commit hooks etc

NodeJS v12 EOL is 4/30/2022, update all JS projects to have a min spec of NodeJS v14

I wanted to make an official Issue, but this PR thread contains a lot of discussion pertaining to the issue and possible solutions:
#930

The "tasks" folder should be named "task"

When running jx pipeline import --url https://github.com/jenkins-x/jx3-pipeline-catalog

I had this error:

loading tekton tasks from: https://github.com/jenkins-x/jx3-pipeline-catalog ...

error: the catalog does not include a 'task' directory at /var/folders/m4/ytp3wjnn1hlbrxt27d64jc3c0000gn/T/jx-git-658269897

translating TaskSpec to Pod: could not parse reference: uses:jenkins-x/jx3-pipeline-catalog/tasks/git-clone/git-clone-pr.yaml@versionStream.

Hello,

jx import project failed with error below :

error: failed to wait for repository to be setup in lighthouse: failed to find trigger in the lighthouse configuration in ConfigMap config in namespace jx for repository

After some investigation found out .lighthouse/jenkins-x/pullrequest.yaml was failing due to the error :
Message: failed to create task run pod "pullrequest-from-build-pack-b24bb": translating TaskSpec to Pod: could not parse reference: uses:jenkins-x/jx3-pipeline-catalog/tasks/git-clone/git-clone-pr.yaml@versionStream. Maybe invalid TaskSpec

Much obliged.

same wrong image for all charts

icon: https://raw.githubusercontent.com/jenkins-x/jenkins-x-platform/d273e09/images/nodejs.png

and

# Default values for node projects.

should be modified in all repos to match the project type (gradle: java...)

Docs explaining real world scenarios

It would be great if the docs explain real world scenarios like:

1. Integrating security scanners like Clair/Trivy/Anchore. I see that the Tekton catalog has a trivy-scanner but how does one go about using it. The only thing I found was "jx pipeline import" which pulled in the trivy-scanner. No clear documentation on the full steps.

2. Build and include other dependant micro services (not databases and the like) in a preview environment as part of an integration test.

There seems to be no books on JenkinsX v3 or courses. Quite a few using v2. The main docs doesn’t go into detail. Love the approach of JenkinsX but the lack of docs is frustrating.

Evaluating JenkinsX for a new AI SaaS product as JX seems to eliminate a lot of plumbing. This would be deployed on Amazon EKS.

Any advice would be appreciated. Thx

PipelineRun timeout not working as expected

Hi,
I'm trying to reduce the timeout of the tasks (in my case, maven-pullrequest), but if I reduce the timeout for testing, from "12h0m0s" to "0h0m30s", the pipeline didn't stop after 30 secs, it still running for 4min.

In my case, I have forked this repository and with "image: uses:lighthouse:[myrepository]/tasks/maven-java11/
pullrequest.yaml@versionStream", I'm able to use my forked tasks (this works fine), but I can't set other timeout...

Also, I have tried to use the timeout inside the task steps, to set custom timeouts to each step, with the same result... Example
For me, this didn't work neither example

Tried this format: "options: timeout: [ time: / unit: ]" and didn't works. I suppose it is for jenkinsX 2...

Im using JenkinsX 3.

Thanks you in advance.
Cumal.

implement git shallow clone for the git-clone task

I'm facing issues with a repository that has a very long history, even though it is not heavy by itself.
As the git-clone task is implemented, every time there is a clone action, the cluster clones the entire repository, which is not efficien; especially when there are large repos (due to long histories).
I propose to implement git shallow clone for the git clone task, instead of the regular clone

proposal:
git clone --depth=1 $REPO_URL $SUBDIR

Original:
git clone $REPO_URL $SUBDIR