stackrox / admission-controller-webhook-demo

Kubernetes admission controller webhook example

Kubernetes Admission Controller Webhook Demo

This repository contains a small HTTP server that can be used as a Kubernetes MutatingAdmissionWebhook.

The logic of this demo webhook is fairly simple: it enforces more secure defaults for running containers as non-root user. While it is still possible to run containers as root, the webhook ensures that this is only possible if the setting runAsNonRoot is explicitly set to false in the securityContext of the Pod. If no value is set for runAsNonRoot, a default of true is applied, and the user ID defaults to 1234.

Prerequisites

A cluster on which this example can be tested must be running Kubernetes 1.9.0 or above, with the admissionregistration.k8s.io/v1beta1 API enabled. You can verify that by observing that the following command produces a non-empty output:

kubectl api-versions | grep admissionregistration.k8s.io/v1beta1

In addition, the MutatingAdmissionWebhook admission controller should be added and listed in the admission-control flag of kube-apiserver.

For building the image, GNU make and Go are required.

Deploying the Webhook Server

1. Bring up a Kubernetes cluster satisfying the above prerequisites, and make sure it is active (i.e., either via the configuration in the default location, or by setting the KUBECONFIG environment variable).
2. Run ./deploy.sh. This will create a CA, a certificate and private key for the webhook server, and deploy the resources in the newly created webhook-demo namespace in your Kubernetes cluster.

Verify

1. The webhook-server pod in the webhook-demo namespace should be running:

$ kubectl -n webhook-demo get pods
NAME                             READY     STATUS    RESTARTS   AGE
webhook-server-6f976f7bf-hssc9   1/1       Running   0          35m

2. A MutatingWebhookConfiguration named demo-webhook should exist:

$ kubectl get mutatingwebhookconfigurations
NAME           AGE
demo-webhook   36m

3. Deploy a pod that neither sets runAsNonRoot nor runAsUser:

$ kubectl create -f examples/pod-with-defaults.yaml

Verify that the pod has default values in its security context filled in:

$ kubectl get pod/pod-with-defaults -o yaml
...
  securityContext:
    runAsNonRoot: true
    runAsUser: 1234
...

Also, check the logs that the pod had in fact been running as a non-root user:

$ kubectl logs pod-with-defaults
I am running as user 1234

4. Deploy a pod that explicitly sets runAsNonRoot to false, allowing it to run as the root user:

$ kubectl create -f examples/pod-with-override.yaml
$ kubectl get pod/pod-with-override -o yaml
...
  securityContext:
    runAsNonRoot: false
...
$ kubectl logs pod-with-override
I am running as user 0

5. Attempt to deploy a pod that has a conflicting setting: runAsNonRoot set to true, but runAsUser set to 0 (root). The admission controller should block the creation of that pod.

$ kubectl create -f examples/pod-with-conflict.yaml 
Error from server (InternalError): error when creating "examples/pod-with-conflict.yaml": Internal error
occurred: admission webhook "webhook-server.webhook-demo.svc" denied the request: runAsNonRoot specified,
but runAsUser set to 0 (the root user)

Build the Image from Sources (optional)

An image can be built by running make. If you want to modify the webhook server for testing purposes, be sure to set and export the shell environment variable IMAGE to an image tag for which you have push access. You can then build and push the image by running make push-image. Also make sure to change the image tag in deployment/deployment.yaml.template, and if necessary, add image pull secrets.

admission-controller-webhook-demo's Issues

example for validation webhook

@misberner

Could you share some examples of the validation webhook as well? I am looking for a sample code to block exec into the pod? Thanks in advance.

Does not working with v1.23

I tested out with the Kubernetes version 1.23 and didn't work. Could you please check this?
Note: - MutatingWebhookConfiguration API version has been changed "admissionregistration.k8s.io/v1beta1" to "admissionregistration.k8s.io/v1".

Regards,

Add clean-up script in Makefile

I think it'll be easier to "play" with that over time if it had a clean command in the Makefile that cleans all the resources it created at once.
I don't mind adding one myself in a PR just want to verify its a good idea? :)

Does not work on Kubernetes 1.19

I went through the setup, when creating a pod, got this error, and the pod did not get mutated:

2020/10/02 00:29:59 http: TLS handshake error from 10.244.0.1:16521: remote error: tls: bad certificate

Getting "Internal error occurred: failed calling webhook "webhook-server.webhook-demo.svc"" error

Hello!

Thanks for prepare this example for me. I'm testing this on GKE and ran into this problem where the webhook service isn't reachable.

[root@gke-client-tf admission-controller-webhook-demo]# k create -f examples/pod-with-override.yaml -n webhook-demo
Error from server (InternalError): error when creating "examples/pod-with-override.yaml": Internal error occurred: failed calling webhook "webhook-server.webhook-demo.svc": Post https://webhook-server.webhook-demo.svc:443/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I struggled at finding the real root cause at first since we have failurePolicy: Ignore in the yaml. I had to update it into failurePolicy: Fail to eventually get where I'm now.

Here's what I've tried

get the pod ip of webhook service

[root@gke-client-tf leilichao]# k describe svc webhook-server -n webhook-demo
Name:              webhook-server
Namespace:         webhook-demo
Labels:            <none>
Annotations:       <none>
Selector:          app=webhook-server
Type:              ClusterIP
IP:                10.10.11.218
Port:              <unset>  443/TCP
TargetPort:        8443/TCP
Endpoints:         10.1.0.168:8443
Session Affinity:  None
Events:            <none>

10.1.0.168 is the pod IP

luanch a busybox pod

I launched a pod in the same namespace as the webhook and curl from it with the

[root@gke-client-tf leilichao]# kubectl -n webhook-demo run curl --image=radial/busyboxplus:curl -i --tty
If you don't see a command prompt, try pressing enter.
[ root@curl:/ ]$ hostname -i
10.1.0.37
[ root@curl:/ ]$ curl -k 10.1.0.168:8443

[ root@curl:/ ]$ curl -k 10.1.0.168:8443

[ root@curl:/ ]$ nslookup webhook-server.webhook-demo.svc
Server:    10.10.11.10
Address 1: 10.10.11.10 kube-dns.kube-system.svc.cluster.local

Name:      webhook-server.webhook-demo.svc
Address 1: 10.10.11.218 webhook-server.webhook-demo.svc.cluster.local
[ root@curl:/ ]$ ping webhook-server.webhook-demo.svc.cluster.local
PING webhook-server.webhook-demo.svc.cluster.local (10.10.11.218): 56 data bytes
^C
--- webhook-server.webhook-demo.svc.cluster.local ping statistics ---
6 packets transmitted, 0 packets received, 100% packet loss
[ root@curl:/ ]$ nslookup webhook-server.webhook-demo.svc.cluster.local
Server:    10.10.11.10
Address 1: 10.10.11.10 kube-dns.kube-system.svc.cluster.local

Name:      webhook-server.webhook-demo.svc.cluster.local
Address 1: 10.10.11.218 webhook-server.webhook-demo.svc.cluster.local
[ root@curl:/ ]$ ping 10.10.11.218
PING 10.10.11.218 (10.10.11.218): 56 data bytes
^C
--- 10.10.11.218 ping statistics ---
11 packets transmitted, 0 packets received, 100% packet loss
[ root@curl:/ ]$ ping 10.1.0.168
PING 10.1.0.168 (10.1.0.168): 56 data bytes
64 bytes from 10.1.0.168: seq=0 ttl=63 time=0.211 ms
64 bytes from 10.1.0.168: seq=1 ttl=63 time=0.126 ms
64 bytes from 10.1.0.168: seq=2 ttl=63 time=0.086 ms
64 bytes from 10.1.0.168: seq=3 ttl=63 time=0.074 ms
64 bytes from 10.1.0.168: seq=4 ttl=63 time=0.084 ms
^C
--- 10.1.0.168 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.074/0.116/0.211 ms

A few observation here:

1. the curl pod is launched at pod ip 10.1.0.37
2. the FQDN webhook-server.webhook-demo.svc.cluster.local is able to be resolved by DNS
3. the connection to 10.1.0.168 works
4. the connection to 10.10.11.218 does not work(of course)

see the webhook pod log from another session

You can see the traffic is comming thru

[root@gke-client-tf admission-controller-webhook-demo]# k get pods -n webhook-demo
NAME                              READY   STATUS    RESTARTS   AGE
curl                              1/1     Running   0          54m
webhook-server-6696bf7b88-gnlkr   1/1     Running   0          8h
[root@gke-client-tf admission-controller-webhook-demo]# k logs -n webhook-demo webhook-server-6696bf7b88-gnlkr
2020/07/20 13:30:43 http: TLS handshake error from 10.1.0.37:36756: tls: first record does not look like a TLS handshake
2020/07/20 13:30:59 http: TLS handshake error from 10.1.0.37:36812: tls: first record does not look like a TLS handshake

Can confirm the traffic went thru from curl pod

FYI I'm on GKE

Now that can confirm the pod ip is reachable and DNS works fine, I suspect there's something wrong with the kube-proxy. However I'm on GKE where the master node isn't accessable so I can't check the kube-proxy for details.

[root@gke-client-tf leilichao]# gcloud container clusters describe my-private-cluster --zone europe-west2-c
addonsConfig:
  kubernetesDashboard:
    disabled: true
  networkPolicyConfig: {}
autoscaling: {}
binaryAuthorization: {}
clusterIpv4Cidr: 10.1.0.0/16
createTime: '2020-06-24T10:59:41+00:00'
currentMasterVersion: 1.15.11-gke.15
currentNodeCount: 3
currentNodeVersion: 1.15.11-gke.15

I'm on 1.15.11-gke.15

Could you please help advice where I could check to fix this issue?

Thanks in advance!

typo in README.md

In the Verify section item 5:
Attempt to deploy a pod that has a conflicting setting: runAsNonRoot set to true, but runAsUser set to false.
"runAsUser" is set to 0 - the user ID not "false".
This can be verified in the pod-with-conflict.yaml file:
securityContext:
runAsNonRoot: true
runAsUser: 0
At any rate, thanks for the great example!

Timeout error

Hi there,

When I ran deploy.sh, it created the deployment but not the pod.

kubectl -n webhook-demo get events

LAST SEEN   TYPE      REASON              KIND         MESSAGE
25m         Warning   FailedCreate        ReplicaSet   Error creating: Timeout: request did not complete within requested timeout 30s
17m 

kubectl -n webhook-demo get deployments

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
webhook-server   0/1     0            0           20m

kubectl -n webhook-demo describe deployment webhook-server

Name:                   webhook-server
Namespace:              webhook-demo
CreationTimestamp:      Fri, 13 Sep 2019 02:11:36 -0400
Labels:                 app=webhook-server
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=webhook-server
Replicas:               1 desired | 0 updated | 0 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=webhook-server
  Containers:
   server:
    Image:        admissioncontrollerwebhook2:v1.0.0
    Port:         8443/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:
      /run/secrets/tls from webhook-tls-certs (ro)
  Volumes:
   webhook-tls-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  webhook-server-tls
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    False   ProgressDeadlineExceeded
OldReplicaSets:  <none>
NewReplicaSet:   webhook-server-7c6767cc65 (0/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  22m   deployment-controller  Scaled up replica set webhook-server-7c6767cc65 to 1

kubectl -n webhook-demo describe rs webhook-server-7c6767cc65

Name:           webhook-server-7c6767cc65
Namespace:      webhook-demo
Selector:       app=webhook-server,pod-template-hash=7c6767cc65
Labels:         app=webhook-server
                pod-template-hash=7c6767cc65
Annotations:    deployment.kubernetes.io/desired-replicas: 1
                deployment.kubernetes.io/max-replicas: 2
                deployment.kubernetes.io/revision: 1
Controlled By:  Deployment/webhook-server
Replicas:       0 current / 1 desired
Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=webhook-server
           pod-template-hash=7c6767cc65
  Containers:
   server:
    Image:        admissioncontrollerwebhook2:v1.0.0
    Port:         8443/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:
      /run/secrets/tls from webhook-tls-certs (ro)
  Volumes:
   webhook-tls-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  webhook-server-tls
    Optional:    false
Events:
  Type     Reason        Age   From                   Message
  ----     ------        ----  ----                   -------
  Warning  FailedCreate  24m   replicaset-controller  Error creating: Timeout: request did not complete within requested timeout 30s