cert-manager / release

Releasing tooling for the cert-manager project

cert-manager

cert-manager adds certificates and certificate issuers as resource types in Kubernetes clusters, and simplifies the process of obtaining, renewing and using those certificates.

It supports issuing certificates from a variety of sources, including Let's Encrypt (ACME), HashiCorp Vault, and Venafi TPP / TLS Protect Cloud, as well as local in-cluster issuance.

cert-manager also ensures certificates remain valid and up to date, attempting to renew certificates at an appropriate time before expiry to reduce the risk of outages and remove toil.

Documentation

Documentation for cert-manager can be found at cert-manager.io.

For the common use-case of automatically issuing TLS certificates for Ingress resources, see the cert-manager nginx-ingress quick start guide.

For a more comprehensive guide to issuing your first certificate, see our getting started guide.

Installation

Installation is documented on the website, with a variety of supported methods.

Developing cert-manager

We actively welcome contributions and we support both Linux and macOS environments for development.

Different platforms have different requirements; we document everything on our Building cert-manager website page.

Note in particular that macOS has several extra requirements, to ensure that modern tools are installed and available. Read the page before getting started!

Troubleshooting

If you encounter any issues whilst using cert-manager, we have a number of ways to get help:

• A troubleshooting guide on our website.
• Our official Kubernetes Slack channel - the quickest way to ask! (#cert-manager and #cert-manager-dev)
• Searching for an existing issue.

If you believe you've found a bug and cannot find an existing issue, feel free to open a new issue! Be sure to include as much information as you can about your environment.

Community

The cert-manager-dev Google Group is used for project wide announcements and development coordination. Anybody can join the group by visiting here and clicking "Join Group". A Google account is required to join the group.

Meetings

We have several public meetings which any member of our Google Group is more than welcome to join!

Check out the details on our website. Feel free to drop in and ask questions, chat with us or just to say hi!

Contributing

We welcome pull requests with open arms! There's a lot of work to do here, and we're especially concerned with ensuring the longevity and reliability of the project. The contributing guide will help you get started.

Coding Conventions

Code style guidelines are documented on the coding conventions page of the cert-manager website. Please try to follow those guidelines if you're submitting a pull request for cert-manager.

Importing cert-manager as a Module

⚠️ Please note that cert-manager does not currently provide a Go module compatibility guarantee. That means that most code under pkg/ is subject to change in a breaking way, even between minor or patch releases and even if the code is currently publicly exported.

The lack of a Go module compatibility guarantee does not affect API version guarantees under the Kubernetes Deprecation Policy.

For more details see Importing cert-manager in Go on the cert-manager website.

The import path for cert-manager versions 1.8 and later is github.com/cert-manager/cert-manager.

For all versions of cert-manager before 1.8, including minor and patch releases, the import path is github.com/jetstack/cert-manager.

Security Reporting

Security is the number one priority for cert-manager. If you think you've found a security vulnerability, we'd love to hear from you.

Follow the instructions in SECURITY.md to make a report.

Changelog

Every release on GitHub has a changelog, and we also publish release notes on the website.

History

cert-manager is loosely based upon the work of kube-lego and has borrowed some wisdom from other similar projects such as kube-cert-manager.

_{^{Logo design by Zoe Paterson}}

release's Issues

Incorrect command line help: should include a --branch argument

./cmrel stage --branch=release-1.0 --release-version=v1.0.2 --help
The stage command will build and stage a cert-manager release to a
Google Cloud Storage bucket. It will create a Google Cloud Build job
which will run a full cross-build and publish the artifacts to the
staging release bucket.

Usage:
  cmrel stage [flags]

Examples:

To stage a release of the 'master' branch to the default staging bucket, run: 

	cmrel stage --git-ref=master

To stage a release of the 'release-0.14' branch to the default staging bucket,
overriding the release version as 'v0.14.0', run:

	cmrel stage --git-ref=release-0.14 --release-version=v0.14.0

Flags:
      --branch string                 The git branch to build the release from. If --git-ref is not specified, the HEAD of this branch will be looked up on GitHub. (default "master")
      --bucket string                 The name of the GCS bucket to stage the release to. (default "cert-manager-release")
      --cloudbuild string             The path to the cloudbuild.yaml file used to perform the cert-manager crossbuild. The default value assumes that this tool is run from the root of the release repository. (default "./gcb/stage/cloudbuild.yaml")
      --git-ref string                The git commit ref of cert-manager that should be staged.
  -h, --help                          help for stage
      --org string                    Name of the GitHub org to fetch cert-manager sources from. (default "jetstack")
      --project string                The GCP project to run the GCB build jobs in. (default "cert-manager-release")
      --published-image-repo string   The docker image repository set when building the release. (default "quay.io/jetstack")
      --release-version string        Optional release version override used to force the version strings used during the release to a specific value.
      --repo string                   Name of the GitHub repo to fetch cert-manager sources from. (default "cert-manager")

Global Flags:
      --debug   If true, output from sub-commands will be directly piped to stderr.

 ./cmrel stage --git-ref=release-1.0 --release-version=v1.0.2
2020/09/22 14:13:51 Root options:
2020/09/22 14:13:51   Debug: false
2020/09/22 14:13:51 Stage options:
2020/09/22 14:13:51   Bucket: "cert-manager-release"
2020/09/22 14:13:51   Org: "jetstack"
2020/09/22 14:13:51   Repo: "cert-manager"
2020/09/22 14:13:51   Branch: "master"
2020/09/22 14:13:51   GitRef: "release-1.0"
2020/09/22 14:13:51   CloudBuildFile: "./gcb/stage/cloudbuild.yaml"
2020/09/22 14:13:51   Project: "cert-manager-release"
2020/09/22 14:13:51   ReleaseVersion: "v1.0.2"
2020/09/22 14:13:51   PublishedImageRepo: "quay.io/jetstack"
2020/09/22 14:13:51 ---
Error: required flag(s) "branch" not set
required flag(s) "branch" not set

cmrel: add check to ensure `--release-version` is semver compliant

To avoid mistakes in future where we accidentally build releases with non-semver versions, we should add a check into the cmrel tool to ensure:

1. --release-version, if set, is semver
2. A 'staged' release, when 'published', also has a release version that is semver (i.e. in the 'release validation' stage)

ref cert-manager/cert-manager#2862

Error response from daemon: manifest for l.gcr.io/google/bazel:4.2.1 not found

While attempting to release cert-manager 1.6.0-alpha.0 I got this error while running cmrel stage:

Error response from daemon: manifest for l.gcr.io/google/bazel:4.2.1 not found: manifest unknown: Failed to fetch "4.2.1" from request "/v2/google/bazel/manifests/4.2.1".

https://console.cloud.google.com/cloud-build/builds/6a5c4888-aea5-4268-9aa5-fda2e45aed8a;step=2?project=cert-manager-release

Maybe since:

• #44

Move cert-manager-release infrastructure to CNCF's GCP account

CNCF have offered for us to move our cert-manager-release GCP project to live under the CNCF's GCP account. cert-manager-release is currently Jetstack internal.

We can migrate you to the CNCF GCP account where a dedicated project will be created for you (actually, if you have one - we’ll migrate it). To proceed, please add me (email removed) as an owner and I’ll take it from there.
CNCF will cover [the costs], no worries
[Permissions and resources should remain unchanged] but if not - I can easily add all the necessary folks there

This issue is to track that migration, and to provide a place for any discussion relating to that move.

go get cmrel fails

Cannot build cmrel using go get github.com/cert-manager/release/cmd/cmrel@latest🙁

% go get github.com/cert-manager/release/cmd/cmrel@latest
go: found github.com/cert-manager/release/cmd/cmrel in github.com/cert-manager/release v0.0.0-20200707132203-ee32c3f05574
go: finding module for package google.golang.org/grpc/naming
go: finding module for package google.golang.org/grpc/naming
../../go/pkg/mod/google.golang.org/[email protected]/internal/pool.go:10:2: module google.golang.org/grpc@latest found (v1.35.0), but does not contain package google.golang.org/grpc/naming

Set up periodic job to publish an experimental release build

So that we have a signal on release builds ahead of release time when things get a bit more busy, we should have a periodic job that stages/mocks a release at least every day.

This will give us an early warning system if issues begin to arise due to changes in the main cert-manager repo.

Cmctl isn't in the main repository anymore

While releasing 1.15.0-alpha.0, I hit the following issue:

https://console.cloud.google.com/cloud-build/builds/78e0c8c3-c8b9-403a-9df8-e89bd4c2eba0?project=cert-manager-release

Step #3: 2024/03/29 10:37:22 Release validation failed:
Step #3: 2024/03/29 10:37:22   - No ctl binaries found in release - this is probably an error!

The cmctl binaries should only be pushed for cert-manager 1.14 and below, not for 1.15 and above.

Create cert-manager specific testing infrastructure

To prepare for the cert-manager release automation drive and the separation of cert-manager prow from jetstack prow, we should have automated cert-manager test / release infrastructure in place.

Progress

• Investigate what we currently have
• Try using official GKE terraform modules - dislike the opinionated setup and resulting cost
• Infrastructure as code - k8s cluster
  • Investigate best practice for GKE
  • Deploy / destroy test cluster with tf
  • Investigate OIDC on GKE - kube-oidc-proxy + gangway + dex + github oauth
  • integrate OIDC into build.
• deployments as code - identity provider for k8s api access (kubectl) deployed
• integrate identity provider with Github - org membership grants read access

Next steps

• Create the repository https://github.com/cert-manager/infrastructure
• Push all config for infra in to the https://github.com/cert-manager/infrastructure
• Port prow jobs to new testing repo under cert-manager org https://github.com/cert-manager/testing
• Create github bot account (in CNCF 1Password)
• Run the new infra
• Decommission old prow setup

Runtime nil pointer exception

Running:

cmrel publish --nomock --release-name "$CMREL_RELEASE_NAME"
2021/05/28 14:48:45 Root options:
2021/05/28 14:48:45   Debug: false
2021/05/28 14:48:45 Publish options:
2021/05/28 14:48:45   Bucket: "cert-manager-release"
2021/05/28 14:48:45   ReleaseName: "v1.4.0-beta.0-528305b5edd3e582a0996d20556a33d5b795564a"
2021/05/28 14:48:45   CloudBuildFile: "./gcb/publish/cloudbuild.yaml"
2021/05/28 14:48:45   Project: "cert-manager-release"
2021/05/28 14:48:45   NoMock: true
2021/05/28 14:48:45   PublishedImageRepo: "quay.io/jetstack"
2021/05/28 14:48:45   PublishedHelmChartGitHubRepo: "jetstack-charts"
2021/05/28 14:48:45   PublishedHelmChartGitHubOwner: "jetstack"
2021/05/28 14:48:45   PublishedHelmChartGitHubBranch: "main"
2021/05/28 14:48:45   PublishedGitHubOrg: "jetstack"
2021/05/28 14:48:45   PublishedGitHubRepo: "cert-manager"
2021/05/28 14:48:45 ---
2021/05/28 14:48:45 Release with version "v1.4.0-beta.0" (528305b5edd3e582a0996d20556a33d5b795564a) will be published
2021/05/28 14:48:45 DEBUG: Loading cloudbuild.yaml file from "./gcb/publish/cloudbuild.yaml"
2021/05/28 14:48:45 DEBUG: building google cloud build API client
2021/05/28 14:48:45 Submitting GCB build job...
2021/05/28 14:48:46 DEBUG: decoding build operation metadata
2021/05/28 14:48:46 ---
2021/05/28 14:48:46 Submitted publish job with name: "a4c2d1e5-f128-4321-8254-94f36dc87df3"
2021/05/28 14:48:46   View logs at: https://console.cloud.google.com/cloud-build/builds/a4c2d1e5-f128-4321-8254-94f36dc87df3?project=1021342095237
2021/05/28 14:48:46   Log bucket: gs://1021342095237.cloudbuild-logs.googleusercontent.com

Threw a nil pointer exception in CloudBuild here.

It appears to be coming from here.

Perhaps something related to the Github user that CloudBuild is/is not using at that time
/cc @wallrj

Design for partial automation of release process

At the moment our release process is largely manual and requires individuals to look at periodic test results in TestGrid to detemine CI health and then perform steps to stage an publish a release.

This does not work that well in practice - lots of toil and risk of human error when looking at TestGrid.

It would be good if it was possible to automate the gating of release on CI health as well as parts of release process

See also https://kubernetes.slack.com/archives/CDEQJ0Q8M/p1655221978315829

Publish latest release number as part of creating a final release

See cert-manager/website#690 (comment) for context.

It would be good if, as part of cutting a new release that is meant to be the latest final (non-alpha/beta release) we could also publish somewhere (probably a txt file in a bucket) the latest release version (i.e v.1.5.3) so we can then use that in our installation instructions (see i.e kubectl install instructions where version number is read from https://dl.k8s.io/release/stable.txt).

Publish SHA256 sums for kubectl plugins

With the new release of the kubectl plugins, it will be interesting to distributing templates on krew index.

One requirement about publishing the plugins in krew is to provide a sha256 sum for every plugin: https://krew.sigs.k8s.io/docs/developer-guide/plugin-manifest/#specifying-plugin-download-options.

cc @munnerz

Build and release kubectl plugin for Apple Silicon

kubectl itself is now built and released for Apple Silicon kubernetes/kubernetes#97743.

We already have darwin/arm in the list of client platform in cert-manager codebase here so this would probably be just a matter of adding it to the release tooling here-ish and finding someone with an M1 Mac willing to test the new binary.

Move the manual steps of our release process to cmrel commands

As described in "deliverable 1", we need to rework cmrel so that cmrel is the single command we need for cutting a new release.

Here is the list of manual steps that we want integrated into cmrel:

1. the release notes (currently done by release-notes wrangling that we have to do in our release process)
2. updating or creating the release branch (currently done manually, tricky step),
3. find a way to "chain" cmrel stage, cmrel publish, and cmrel publish --nomock) without having to copy-paste manually the "release name" between each step

This issue is about automating (1), (2), and (3), and to make (4) as easy as possible e.g. just uncheck the "Draft" option in the GitHub release.

Progress as of now:

• create cmrel update-release-branch: ETA 3 days work, 20% done (#36) see #31 (comment)
• create release-notes: ETA 1 day work, 0% done
• automate passing the "release name" between the stage, publish, and publish --nomock steps: ETA 1 day work, 0% done