kimschles / secrets-store-demo

Getting Started with the Kubernetes Secrets Store CSI Driver

In Kubernetes, it can be difficult to keep application API keys, access tokens and passwords safe. There are several different approaches to solving this problem, and in this talk Kim will demonstrate how to install Hashicorp Vault and the Secrets Store CSI Driver so that your applications can access secrets stored in ephemeral volumes.

This tutorial was written for a Data on Kubernetes Talk given on 21 April 2022. Getting Started with the Kubernetes Secrets Store CSI Driver

Prerequisites

• A Kubernetes Cluster using version 1.19+
• kubectl
• Helm 3

Resources

• Secrets Store CSI Driver Documentation
• Kubernetes CSI Developer Documentation
• HashiCorp Learn: Mount Vault Secrets through Container Storage Interface (CSI) Volume
• Kubernetes Auth Method

secrets-store-demo's Issues

random revocation of tokens

I implemented this solution but enabled secret rotation and vault seems to randomly revoke tokens.

vault secrets engine ttl = 14 days
the leas ttl = 14 days
the csidriver yaml

    requiresRepublish: true
    storageCapacity: false
    tokenRequests:
    - audience: vault
      expirationSeconds: 600
    volumeLifecycleModes:
    - Ephemeral

--rotation-poll-interval=10080m 

Then after about 5 hours the secret remains the same but vault decides to revoke the credential.
To Fix I need to then delete the secret and cycle the pods.

Maybe add some knowledge on how to use this solution with ttl's and secret rotation?

great tutorial btw.

kubectl apply -f manifests/deployment.yaml fails

When executing (Step 7):

kubectl apply -f manifests/deployment.yaml

I get the following error message:

The Pod "busybox" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)
  core.PodSpec{
  	Volumes:        {{Name: "vault-database", VolumeSource: {CSI: &{Driver: "secrets-store.csi.k8s.io", ReadOnly: &true, VolumeAttributes: {"secretProviderClass": "vault-database"}}}}, {Name: "kube-api-access-hzfc8", VolumeSource: {Projected: &{Sources: {{ServiceAccountToken: &{ExpirationSeconds: 3607, Path: "token"}}, {ConfigMap: &{LocalObjectReference: {Name: "kube-root-ca.crt"}, Items: {{Key: "ca.crt", Path: "ca.crt"}}}}, {DownwardAPI: &{Items: {{Path: "namespace", FieldRef: &{APIVersion: "v1", FieldPath: "metadata.namespace"}}}}}}, DefaultMode: &420}}}},
  	InitContainers: nil,
  	Containers: []core.Container{
  		{
  			... // 5 identical fields
  			Ports:   nil,
  			EnvFrom: nil,
- 			Env: []core.EnvVar{
- 				{
- 					Name:      "API_TOKEN",
- 					ValueFrom: &core.EnvVarSource{SecretKeyRef: &core.SecretKeySelector{...}},
- 				},
- 			},
+ 			Env:          nil,
  			Resources:    {Limits: {s"cpu": {i: {...}, s: "200m", Format: "DecimalSI"}, s"memory": {i: {...}, s: "100Mi", Format: "BinarySI"}}, Requests: {s"cpu": {i: {...}, s: "100m", Format: "DecimalSI"}, s"memory": {i: {...}, s: "50Mi", Format: "BinarySI"}}},
  			VolumeMounts: {{Name: "vault-database", ReadOnly: true, MountPath: "/mnt/secrets-store"}, {Name: "kube-api-access-hzfc8", ReadOnly: true, MountPath: "/var/run/secrets/kubernetes.io/serviceaccount"}},
  			... // 12 identical fields
  		},
  	},
  	EphemeralContainers: nil,
  	RestartPolicy:       "Always",
  	... // 25 identical fields
  }

The deployment.yaml looks like this:

kind: Pod
apiVersion: v1
metadata:
  name: busybox
  namespace: demo
spec:
  serviceAccountName: app-sa
  containers:
  - image: k8s.gcr.io/e2e-test-images/busybox:1.29
    name: busybox
    imagePullPolicy: IfNotPresent
    command:
    - "/bin/sleep"
    - "10000"
    resources:
      requests:
        cpu: 100m
        memory: 50Mi
      limits:
        cpu: 200m
        memory: 100Mi
    volumeMounts:
    - name: vault-database
      mountPath: "/mnt/secrets-store"
      readOnly: true
    # Uncomment after syncing the Vault data with a Kubernetes Secret  
    env:
    - name: API_TOKEN
      valueFrom:
         secretKeyRef:
           name: kvsecret-1
           key: token
  volumes:
    - name: vault-database
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "vault-database"

atael@ATAEL-mac secrets-store-demo % kubectl get pods -n demo
NAME                                               READY   STATUS    RESTARTS   AGE
busybox                                            1/1     Running   0          13m
csi-secrets-store-secrets-store-csi-driver-g6fvn   3/3     Running   0          22m
csi-secrets-store-secrets-store-csi-driver-gkcvm   3/3     Running   0          22m
csi-secrets-store-secrets-store-csi-driver-jrjdm   3/3     Running   0          22m
vault-0                                            1/1     Running   0          37m
vault-csi-provider-2zqb5                           1/1     Running   0          37m
vault-csi-provider-9tnvs                           1/1     Running   0          37m
vault-csi-provider-qdkz2                           1/1     Running   0          37m

atael@ATAEL-mac secrets-store-demo % kubectl get nodes
NAME          STATUS   ROLES   AGE   VERSION
10.0.10.107   Ready    node    12d   v1.22.5
10.0.10.149   Ready    node    12d   v1.22.5
10.0.10.197   Ready    node    12d   v1.22.5