eumel8 / cosignwebhook

Kubernetes Validation Admission Controller to verify Cosign signatures

Cosign Webhook

Kubernetes Validation Admission Controller to verify Cosign Image signatures.

This webhook watches for pod creation in deployments and verifies all container image it finds with an existing RSA public key (if present).

Installation with Helm

helm -n cosignwebhook upgrade -i cosignwebhook oci://ghcr.io/eumel8/charts/cosignwebhook --versi
on 3.0.0 --create-namespace

this installation has some advantages:

• automatic generation of TLS key pair
• automatic setup of ServiceMonitor and Grafana dashboards

If you use your own image, you'll have to sign it first. Don't forget to change the cosign.scwebhook.key value to your public key, used to sign the image.

Installation with manifest

As cluster admin, create a namespace and install the admission controller:

kubectl create namespace cosignwebhook
kubectl -n cosignwebhook apply -f manifests/rbac.yaml
kubectl -n cosignwebhook apply -f manifests/manifest.yaml

The manifest contains a self-signed example ca, TLS certificate, and key. This is only to see how it looks like, you should generate your own certificate, see below:

Cert generation

Run the generate-certs script in the hack folder to generate the TLS key pair and the CA certificate for the webhook:

generate-certs.sh --service cosignwebhook --webhook cosignwebhook --namespace cosignwebhook --secret cosignwebhook

Validating your container images

To use the webhook, you need to first sign your images with cosign, and then use one of the following validation possibilities.

• Public key as environment variable
• Public key as secret reference
• Public key as default secret for namespace

Additionally, if the signature of the image you're trying to validate is not in the same repository as the image, you need to add the COSIGN_REPOSITORY environment variable to the environment of the container:

# in the container spec of the workload
env:
  - name: COSIGN_REPOSITORY
    value: myregistry.io/signatures

This option is similar to the COSIGN_REPOSITORY environment variable used with cosign verify and cosign sign command line tool and is used to specify the repository where the signature of the image is located, if it's not in the same repository as the image.

Public key as environment variable

Add your Cosign public key as env var in container spec of the first container:

env:
  - name: COSIGNPUBKEY
    value: |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEGOrnlJ1lFxAFTY2LF1vCuVHNZr9H
      QryRDinn+JhPrDYR2wqCP+BUkeWja+RWrRdmskA0AffxBzaQrN/SwZI6fA==
      -----END PUBLIC KEY-----

Public key as secret reference

Instead of hardcoding the public key in the deployment, you can also use a secret reference. The key and the secret may be named freely, as long as the secret contains a valid public key.

apiVersion: v1
kind: Secret
data:
  COSIGNPUBKEY: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFS1BhWUhnZEVEQ3ltcGx5emlIdkJ5UjNxRkhZdgppaWxlMCtFMEtzVzFqWkhJa1p4UWN3aGsySjNqSm5VdTdmcjcrd05DeENkVEdYQmhBSTJveE1LbWx3PT0KLS0tLS1FTkQgUFVCTElDIEtFWS0tLS0t
metadata:
  name: cosign_secret # Can be choosen freely
type: Opaque

        env:
          - name: COSIGNPUBKEY
            valueFrom:
              secretKeyRef:
                name: cosign_secret # Must be equal to metadata.name of the secrect
                key: COSIGNPUBKEY

Public key as default secret for namespace

Create a default secret for all your images in a namespace, which the webhook will always search for, when validating images in this namespace:

apiVersion: v1
kind: Secret
data:
  COSIGNPUBKEY: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFS1BhWUhnZEVEQ3ltcGx5emlIdkJ5UjNxRkhZdgppaWxlMCtFMEtzVzFqWkhJa1p4UWN3aGsySjNqSm5VdTdmcjcrd05DeENkVEdYQmhBSTJveE1LbWx3PT0KLS0tLS1FTkQgUFVCTElDIEtFWS0tLS0t
metadata:
  name: cosignwebhook
type: Opaque

The name of the secret must be cosignwebhook and the key COSIGNPUBKEY. The value of COSIGNPUBKEY must match the public key used to sign the image you're deploying.

Test

To test the webhook, you may run the following command(s):

# unit tests
make test-unit

# E2E tests
make e2e-prep
make test-e2e

E2E tests

The E2E tests require a running kubernetes cluster. Currently, the namespace and webhook are deployed via helper make targets. To run the tests the following is required:

• docker
• cosign (v2)

To run the E2E tests, the following steps are required (in order):

• create a k3d local cluster for the tests and a local iamge registry (make e2e-cluster)
• signing keys are generated (make e2e-keys)
• a new cosignwebhook image is build and signed with a temp key (make e2e-images)
• the image is pushed to a local registry & deployed to the test cluster (make e2e-deploy)

To do all of the above, simply run make e2e-prep. Each step should also be able to be executed individually. To clean up the E2E setup, run make e2e-cleanup. This will delete everything created by the E2E preparation. If you've already created the cluster and the keys, and you're actively testing new code, you may run make e2e-images e2e-deploy test-e2e to test your changes.

Local build

CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' -o cosignwebhook

Credits

• Bruno Bressi [email protected]
• Frank Kloeker [email protected]

Life is for sharing. If you have an issue with the code or want to improve it, feel free to open an issue or an pull request.

The Operator is inspired by @pipo02mix, a good place to learn fundamental things about Admission Controllert

cosignwebhook's Issues

Secret(k8s) cannot be used to define COSIGNPUBKEY

Current behaviour:
If I define Secret(k8s) and use it to define Pod(k8s) environment variable COSIGNPUBKEY (env[].valueFrom.secretKeyRef), Webhook does not take Secret(k8s) value.

Expected behaviour:
I am able to define COSIGNPUBKEY value using Secret(k8s).

The manifest deployment is outdated

We still offer the possibility to install cosignwebhook without Helm. This manifests are outdated:

• image version old
• mentioned cosign pub keys are wrong
• rbac missing permissions for EventRecorder
• demoapp won't start with shared secret and different signed images

Helm Upgrade deletes certificate in secret and validationwebhook

Support COSIGN_REPOSITORY variable to allow signatures to be read from other image repositories

Background

Per default, cosign searches for the image's signature in the same repository as the image itself. This is unfortunately not always the case with some images (bitnami images are an example of this). Cosign offers the option of pushing the signature of the image to a separate repository, which can be defined using the COSIGN_REPOSITORY environmental variable, when using the CLI to sign the image.

Users of the webhook should be able to allow such images to also be verified, when they provide the public key and the signature repository.

More details

Internally, cosign creates an additional ociremote.Option with ociremote.WithTargetRepository to override the repository, in which the signature of the image will reside.

Proposal

Allow users to define an environment variable per container (COSIGN_REPOSITORY), which will be used when calling cosign.VerifyImageSignatures to instruct cosign which repository to choose for the signature validation.

Kube-API client rate limiter Wait returned an error: context canceled

If a user has multiple like many secretRef in the deployment manifest, the KubeAPI can be overloaded due the many of requests. The list should be count or limited per time

Change Log Service

At the moment logs are pushed to STDOUT in a timestamp format whithout date and stored on the disk under /tmp
Change the timestamp format and don't store data on the disk.

Fix code scanning alert - exported method CosignServerHandler.Healthz should have comment or be unexported

Tracking issue for:

• https://github.com/eumel8/cosignwebhook/security/code-scanning/20

all images needs to sign while cosignwebhook expects signatures

version 4.0.3

at the moment all container and init-container (also injected init-container by istio, dynatrace) needs to sign, otherwise cosignwebhok will deny the Admission request. Also different keys seems not possible.

needs to test further

cc. @puffitos

panic on getPubKeyFromEnv

cosignwebhook:3.2.0

cosignwebhook crash in getPubKeyFromEnv:

2023/11/06 08:06:30 http: panic serving 10.42.0.0:8578: runtime error: invalid memory address or nil pointer derefere
nce
goroutine 10112 [running]:
net/http.(*conn).serve.func1()
        /usr/local/go/src/net/http/server.go:1868 +0xb9
panic({0x1ce8a00?, 0x326cae0?})
        /usr/local/go/src/runtime/panic.go:920 +0x270
main.(*CosignServerHandler).getPubKeyFromEnv(0xc000656000?, 0xc000ba0900)
        /app/cosignwebhook.go:110 +0xa1
main.(*CosignServerHandler).serve(0xc0004bc870?, {0x22dc190, 0xc0009280e0}, 0xc00027aa00?)
        /app/cosignwebhook.go:183 +0x366
net/http.HandlerFunc.ServeHTTP(0x10?, {0x22dc190?, 0xc0009280e0?}, 0xc0003b23ec?)
        /usr/local/go/src/net/http/server.go:2136 +0x29
net/http.(*ServeMux).ServeHTTP(0x410725?, {0x22dc190, 0xc0009280e0}, 0xc00027aa00)
        /usr/local/go/src/net/http/server.go:2514 +0x142
net/http.serverHandler.ServeHTTP({0x22d14e0?}, {0x22dc190?, 0xc0009280e0?}, 0x6?)
        /usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc000967290, {0x22e9a58, 0xc00062e270})
        /usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 13
        /usr/local/go/src/net/http/server.go:3086 +0x5cb

The last validated Pod comes from a Job and has no Env var and no secret.

cc: @puffitos

TLS error if the scratch image is used

InScratchthere is no certificate bundle installed, so outside connection will fail:

2023/01/06 17:32:36 http: TLS handshake error from 10.42.0.0:21495: remote error: tls: bad certificate

Improve installation procedure

At the moment cosignwebhook won't start on first installation while failurePolicy: Fail while service is not available, only if failurePolicy: Ignore is set. Better is to install the app, wait until service is available and setup the ValidationWebhook.
Not sure if this is done in a Helm chart.

Only ECDSA keys are accepted - RSA keys should be accepted as well

Describe the problem

A customer is struggling to use CosignWebhook after updating their Cosign key for their images. Event:

[FailedCreate] Internal error occurred: failed calling webhook "cosignwebhook.caas.telekom.de": failed to call webhook: Post "https://cosignwebhook.cosignwebhook.svc:443/validate?timeout=10s": EOF

They can't use ECDSA keys...

The use of RSA public keys is forbidden; only ECDSA keys are allowed.

cosign logs:

2024/08/13 11:10:11 http: panic serving 10.42.33.0:63420: interface conversion: crypto.PublicKey is *rsa.PublicKey, not *ecdsa.PublicKey

Possible solution

• Allow RSA keys as well

If I understand the code correctly, the ECDSA requirement arises here:

cosignwebhook/webhook/cosignwebhook.go

Line 341 in 92c9f45

cosignLoadKey, err := signature.LoadECDSAVerifier(publicKey.(*ecdsa.PublicKey), crypto.SHA256)

Therefore, this part would need to be adjusted to also accept other PublicKeys.

Loading public key also from secret resource

At the moment it's only possible to load the cosign public key as env var in Kubernetes deployment.
Additional use case would be to store this key in a secret and load it from them:

env:
- name: COSIGNPUBKEY
  valueFrom:
    secretKeyRef:
      key: cosign.pub
      name: cosign 

Successfull image verification not notified to user

If the image is successful verified no status message is notified to user.
An Eventlog entry would be helpful

Feat: Support multi container verification

At the moment, cosignwebhook verified the first container of a POD with these env variable there. Would be good, to verify all existing container and init-container.

Webhook SSL Error after Kubernetes Update

After upgrade the certificates of the Kubernetes Cluster, cosignwebhook denied all workload with

Error creating: Internal error occurred: failed calling webhook "cosignwebhook.caas.telekom.de": failed to call webhook: Post "https://cosignwebhook.cosignwebhook.svc:443/validate?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "cosign-webhook-ca")

EventRecorder always post PodVerified

version: 4.0.2

Just realized that the EventRecorder always puts a message in the event channel if a Pod is create, also if there is no signature and no verification key is present.

60m         Normal    PodVerified         pod/demoapp-bcb9687f9-wdc2h     Signature of pod's images(s) verified successfully

I think this comes from the end of Admission verification process.

accept is the right return after finishing validation process, but there is still the option that we never verified an image

cc @puffitos