bwolmarans / kitty

Enforcing and Observing Signed Deployments with the Kubernetes Admission Controller

Software Supply Chain management in your Kubernetes Cluster is the problem du jour.

But why?

Software Supply Chain Issues are top of mind because our apps aren't entirely ours anymore - it would not be uncommon to have more 3rd (and 4th and 5th and onwards) party code in your apps than code you have written yourself. Once study found that the average application includes 118 libraries.

But what about container respositories themselves, and the moment of truth when we are deploying containers into the cluster in Kubernetes, is there a way to have some semblance of control over that?

Every organization has policies. Some exist to meet legal or governance goals, while others may help ensure best practices and conventions. Approaching compliance manually would be prone to the usual basket of human-error issues and frustrations. If we can automate policy enforcement into the deployment flow, we could achieve consistency, increase development efficiency through immediate feedback, and be more agile by giving developers the freedom to operate independently within a structure of compliance.

Kubernetes allows decoupling policy decisions from the inner workings of the API Server by means of admission controller webhooks, which are executed whenever a resource is created, updated or deleted.

Admission Controllers Overview

Kubernetes cluster Admission Controllers are very useful, and Sysdig has one already and I tried it (below) :+1: This PR looks great - it's ready to merge! :shipit:

And here is what it looks like in the overview:

The Challenge

So to write this blog, I was given some great guidelines, a challenge to embark on:

1. Sign and publish a container image to an OCI registry
2. Demonstrate how the signature verification is performed in the cluster
3. Block signed images in a specific namespace, allow but warn on other namespaces *Big Assumption: Allow signed images in a specific ns, not block
4. Notify of blocked or noncompliant images in Sysdig events UI

Enter the Kubernetes Admmission Controller, a relatively new concept that has come under the spotlight recently due in no small part to the urgency around supply chain attacks. After all, if we can control admission into the cluster, it is the last line of defense.

Integrity and provenance of container images deployed to a Kubernetes cluster can be ensured via digital signatures.

Step 1 of the Challenge

Sign and publish a container image to an OCI registry

Please take a look in this repo at my Github Actions where I build my image,push it to ECR, use cosign to Sign it

Step 2 of the Challenge

Demonstrate how the signature verification is performed in the cluster

*TLDR; I look forward (very, very much) to the day when hopefully soon, image signing and selectively admitting images based on their signing can be as smooth as what we already see today in Admission Controllers such as the one from Sysdig:

So as shown below, we can utilize Cosign and Kyverno. This is the part of the blog where I learn the hard way that Container image signing has been and still is a bit of a gap in the security landscape, and that solutions here are very much in flux. Pardon the pun. It seems that the good folks at Docker Content Trust/Notary came out of the gates with v1, but that never really gained traction, and while v2 is out now, there are multiple options floating around including projects including Kyverno, looks very interesting, it’s still in the design phase (AFAIK).

The Cosign project is part of the Sigstore initiative, so I was interested to take a look at it and see how it works. Sigstore has interesting ideas about software transparency logs, but for this blog, I’ll just be looking at the raw image signing process.

As shown below, we can sign, and verify, and use Kyverno to verify signed images in the cluster.

Step 3 of the Challenge

Block signed images in a specific namespace, allow but warn on other namespaces *Big Assumption: I am making the assumption the intent is not to Block signed images, but to actually Allow signed images, as that would be more logical

Here is where I struggeled.

A Kyverno policy can only Audit or Enforce, but not on multiple namespaces at the same time. I don't think we're there yet, as shown by my demonstration below. So I tried swapping betweed Red and Blue namespaces, with two different Kyverno policies, but as you can see in this screen capture below, it didn't really work.

Hopping on the Kyverno slack channel (a very helpful community) I was told that 1.6 will have per-namespace actions support, but I wasn't brave enough to try 1.6 because it wasn't released yet, and honestly I would need more time than this weekend to test this out.

So I switched between namespace, and, as of VERY RECENTLY managed to meet the requirements, in a fashion, as shown below in Step 4.

I also tried Gatekeeper for OPA but I couldn't find any good examples of using Gatekeeper for verification of signed images. What's worse is installing Gatekeeper failed because I had previously intalled Kyverno, and Gatekeeper actually uses Kyverno under the hood. Simply un-installing Kyverno manually or via the Helm charts didn't work. But after I posted some questions on the OPA Slack channel with my logs, the community came to my rescue and helped my manually install the mutating webooks that my original Kyverno install left behind! After that I was able to get Gatekeepr installed, but by now it was 8PM on Sunday night and I was out of time.

I should probably have tried Connaiseur, but that's for next weekend!

Step 4 of the Challenge

Notify of blocked or noncompliant images in Sysdig events UI

Due to deadlines, and other commitments, while I made an effort to get this part into my github actions, but had to settle for a bash script for the time being.
You can see my script in this repo here: sysdig-interview.sh and note you will need to set the environment var SYSDIG_API_TOKEN to your Sysdig API Token.

And here is what it looks like! Notice we are blocking the doggy-unsigned image.

Conclusion

My conclusion is we need a enterprise, user-friendly solution to this. I would like to see the Sysdig Admission Controller give me a nice way to handle this the same way it support the other features it has today.

I think that would make life easier for me, and all K8S admins!