doitintl / secrets-init

minimalistic init system for containers with AWS/GCP secrets support

Blog Post

Kubernetes and Secrets Management in the Cloud

secrets-init

secrets-init is a minimalistic init system designed to run as PID 1 inside container environments, similar to dumb-init, integrated with multiple secrets manager services:

• AWS Secrets Manager
• AWS Systems Manager Parameter Store
• Google Secret Manager

Why you need an init system

Please read Yelp dumb-init repo explanation

Summary:

• Proper signal forwarding
• Orphaned zombies reaping

What secrets-init does

secrets-init runs as PID 1, acting like a simple init system. It launches a single process and then proxies all received signals to a session rooted at that child process.

secrets-init also passes almost all environment variables without modification, replacing secret variables with values from secret management services.

Integration with AWS Secrets Manager

User can put AWS secret ARN as environment variable value. The secrets-init will resolve any environment value, using specified ARN, to referenced secret value.

If the secret is saved as a Key/Value pair, all the keys are applied to as environment variables and passed. The environment variable passed is ignored unless it is inside the key/value pair.

# environment variable passed to `secrets-init`
MY_DB_PASSWORD=arn:aws:secretsmanager:$AWS_REGION:$AWS_ACCOUNT_ID:secret:mydbpassword-cdma3

# environment variable passed to child process, resolved by `secrets-init`
MY_DB_PASSWORD=very-secret-password

Integration with AWS Systems Manager Parameter Store

It is possible to use AWS Systems Manager Parameter Store to store application parameters and secrets.

User can put AWS Parameter Store ARN as environment variable value. The secrets-init will resolve any environment value, using specified ARN, to referenced parameter value.

# environment variable passed to `secrets-init`
MY_API_KEY=arn:aws:ssm:$AWS_REGION:$AWS_ACCOUNT_ID:parameter/api/key
# OR versioned parameter
MY_API_KEY=arn:aws:ssm:$AWS_REGION:$AWS_ACCOUNT_ID:parameter/api/key:$VERSION

# environment variable passed to child process, resolved by `secrets-init`
MY_API_KEY=key-123456789

Integration with Google Secret Manager

User can put Google secret name (prefixed with gcp:secretmanager:) as environment variable value. The secrets-init will resolve any environment value, using specified name, to referenced secret value.

# environment variable passed to `secrets-init`
MY_DB_PASSWORD=gcp:secretmanager:projects/$PROJECT_ID/secrets/mydbpassword
# OR versioned secret (with version or 'latest')
MY_DB_PASSWORD=gcp:secretmanager:projects/$PROJECT_ID/secrets/mydbpassword/versions/2

# environment variable passed to child process, resolved by `secrets-init`
MY_DB_PASSWORD=very-secret-password

Project auto-detection

If secret-manager is running in an environment where the Google metadata server is available, or the -google-project flag is set, the secret path may be omitted, and the current project is used.

MY_DB_PASSWORD=mydbpassword
MY_DB_PASSWORD=mydbpassword/versions/2

Requirement

Container

If you are building a Docker container, make sure to include the ca-certificates package, or use already prepared doitintl/secrets-init Docker container (linux/amd64, linux/arm64).

AWS

In order to resolve AWS secrets from AWS Secrets Manager and Parameter Store, secrets-init should run under IAM role that has permission to access desired secrets.

This can be achieved by assigning IAM Role to Kubernetes Pod or ECS Task. It's possible to assign IAM Role to EC2 instance, where container is running, but this option is less secure.

Google Cloud

In order to resolve Google secrets from Google Secret Manager, secrets-init should run under IAM role that has permission to access desired secrets.

This can be achieved by assigning IAM Role to Kubernetes Pod with Workload Identity. It's possible to assign IAM Role to GCE instance, where container is running, but this option is less secure.

Kubernetes secrets-init admission webhook

The kube-secrets-init implements Kubernetes admission webhook that injects secrets-init initContainer into any Pod that references cloud secrets (AWS Secrets Manager, AWS SSM Parameter Store and Google Secrets Manager) implicitly or explicitly.

Code Reference

Initial init system code was copied from go-init project.

secrets-init's Issues

failed to get secret from AWS Parameters Store: WebIdentityErr: failed to retrieve credentials

I installed by this helm chart kube-secrets-init and enble cert-manager = true. All oke but when i create a pod for the test and got log console like:
time="2022-10-27T06:27:00Z" level=error msg="failed to resolve secrets" error="failed to get secret from AWS Parameters Store: WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post \"https://sts.amazonaws.com/\": x509: certificate signed by unknown authority"

In my case:

I install in EKS with version 1.23.
I use ServiceAccount for AssumeRoleWithWebIdentity from Pod.
Anyone know how to resolve this issue?

Option to exit early on error

Hey @alexei-led

Thanks for the project, it's really helpful!

I was wondering if it's possible to have an option/flag to exit non-zero if a secret can't be found/accessed instead of warning and still running the child process? It would be much quicker to debug issues with bad secrets if this can be done.

Thanks in advance!

Do any secrets get stored within a layer of the docker image?

How does secrets-init avoid storing secrets inside the image?

Secrets fail to load half the time

We're using secrets-init in combination with godotenv to source environment variables and then inject secrets into the environment.

We're seeing unexpected behavior where secrets-init does not set environment variables properly roughly half the time.

I also noticed that secrets-init does not overwrite an existing environment variable if it's already set. Is this by design or a bug?

❯ cat .env
ENCRYPTION_KEY=
AWS_REGION=us-west-2
AWS_SECRETS=arn:aws:secretsmanager:us-west-2:123456789012:secret:my-secret

❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=pulled_from_secrets_manager

# 12:36:32
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=

# 12:36:34
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=pulled_from_secrets_manager

# 12:36:35
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=pulled_from_secrets_manager

# 12:36:37
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=

# 12:36:39
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=pulled_from_secrets_manager

Race condition between reaper and command

I observe that sometime secrets-init logs a message level=error msg="failed to wait for command to complete" error="waitid: no child processes" and exits with an error code.

This appears to be because sometimes the subprocess started by exec.Command is reaped by syscall.Wait4 before cmd.Wait reaps it.

I think an alternative implementation could not use cmd.Wait and instead let syscall.Wait4 reap all children, as done by tini.

In addition, I think it would be more useful to return the return code from the subprocess instead of always exiting with code 1 on error.

I hope this is useful! Thanks for the tool.

secrets-init does not overwrite existing empty environment variables

Hey @alexei-led thanks for the project. I noticed confusing behavior that secrets-init doesn't set an environment variable if it already exists as an empty value. Can this behavior be changed or be configurable? Our scenario is that we have default values in our environment but want to overwrite if they have a matching secret in Secrets Manager.

Thanks!

Allow changing log format

It's really annoying when everything logs in JSON, except one thing that needs a specific log parser.

It'd be nice to have an option to change the logging format to JSON. Also, having some sort of identifier where the log comes from (ie. subsystem=secrets-init) would also be nice.

This should also be added to the admission webhook (eg. modify pods to add a SECRETS_INIT_LOG_FORMAT env var as well).

failed to get secret from AWS Secrets Manager: MissingRegion: could not find region configuration

If you don't specify AWS_REGION environment variable, you won't be able to fetch an ASM secret. This could be pulled from the ARN of the secret

Failed to send system signal to the process

The other issue is resolved however I'm getting a new error:

Saras-MBP:deploy sarajarjoura$ kubectl logs -f $POD --all-containers
testing
time="2020-03-30T13:35:59Z" level=error msg="failed to send system signal to the process" args="[sh -c echo $TEST_SECRET]" error="no such process" path=/bin/sh pid=16 signal="urgent I/O condition"

Thanks in advance.

secrets-init fails if google root CA is not installed

Hello,
I'm using busybox and secrets-init 0.4.0 fails if google root CA is not available.

/ # /tmp/bin/secrets-init --provider google env ERRO[0000] failed to resolve secrets error="failed to get secret from Google Secret Manager: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: x509: certificate signed by unknown authority\""

I'm sure if this is by design or if secrets-init should come with a pre-installed CA somehow.

Could you please also document it, maybe in the document below as well in the READM.md?
https://blog.doit-intl.com/kubernetes-and-secrets-management-in-cloud-858533c20dca

Thx!

Docker run not working as expected

I have a Dockerfile which consists of:

ENTRYPOINT [ "/secrets-init", "--provider=google"]
CMD ["php-fpm"]

If I am using my Dockerfile in relation with docker compose it works as expected and variables are being replaced.
Commands like docker compose run test sh -c "echo \$TEST " show the replaced variable.
But when I am using the same image and I use docker run --env-file=.env ... test sh -c "echo \$TEST" , it doesn't show the replaced variable. I'd expect them to behave the same, but there must be a difference regarding the environment. Authentication with Google is working for both commands. Can't I use /secrets-init with docker run ...?

Sometimes I get a "ERRO[0000] failed to send system signal to the process"

secrets-init:0.4.0

Sometimes, I don't know why, I get these errors:
kubectl exec --stdin --tty mypod-init -- /bin/ash Defaulted container "mypod-cm" out of: mypod-cm, writefile (init) / # /tmp/bin/secrets-init --provider google env
GOOGLE_APPLICATION_CREDENTIALS=/credential/credentials.json
PASS=It worked!
ERRO[0000] failed to send system signal to the process args="[env]" error="no such process" path=/usr/bin/env pid=23 signal=SIGURG

secrets-init is doing its job, the PASS was translated. Still, I get the error below sometimes, pretty rare, but it happens.

ERRO[0000] failed to send system signal to the process args="[env]" error="no such process" path=/usr/bin/env pid=23 signal=SIGURG

Panic and Segmentation error when accessing secret via Google Provider on k8s pod

I am using the following deployment file

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: secretmanagerdemo
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: secretmanagerdemo
    spec:
      initContainers:
        - name: secrets-init
          image: doitintl/secrets-init:v0.2.4
          command:
            - sh
          args:
            - -c
            - "cp /usr/local/bin/secrets-init /secrets-init/bin/"
          volumeMounts:
            - mountPath: /secrets-init/bin
              name: secrets-init-volume
      containers:
        - image: alpine:3
          name: print-env
          env:
            - name: TEST_SECRET
              value: gcp:secretmanager:projects/$PROJECT_ID/secrets/testsecret
          command:
            - /secrets-init/bin/secrets-init
          args:
            - --provider
            - google
            - sh
            - -c
            - 'echo $TEST_SECRET'
          volumeMounts:
            - mountPath: /secrets-init/bin
              name: secrets-init-volume
      volumes:
        - name: secrets-init-volume
          emptyDir: {}

I am getting the expected output of the secret value but the container is exiting with an error due to the following:

/$ POD=secretmanagerdemo-6ff8dd4d4b-vgjvm
/$ kubectl logs -f $POD --all-containers
testing
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xad45ad]

goroutine 62 [running]:
main.run.func1(0xc0000ab680, 0xc0000d14a0)
	/home/runner/work/secrets-init/secrets-init/main.go:155 +0xad
created by main.run
	/home/runner/work/secrets-init/secrets-init/main.go:150 +0x363

Any ideas? Thanks in advance

Environment variable value not inherited in child process or application container.

I used below snippet. After pod/container get created, I can manually run aws cli with secrets manager command and it worked so means, service account, aws iam role is correct but still not found variable value in /tmp/xyz as per command or not in log or not when ran manually echo $MY_PASSWORD. What's wrong?

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ekssecrettest
  namespace: dev  
  labels:
    app: ekssecrettest
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ekssecrettest
  template:
    metadata:
      labels:
        app: ekssecrettest
    spec:
      serviceAccountName: ekssecrettest
      restartPolicy: Always      
      initContainers: 
        - name: ekssecrettest
          image: 'doitintl/secrets-init:latest'
          env:
            - name: MY_PASSWORD
              value: arn:aws:secretsmanager:eu-central-1:111111111111:secret:ekssecrettest-5zaSrq    
      containers:
        - name: awscli
          image: 'amazon/aws-cli'
          command: ["/bin/sh","-c"]
          args: ["echo $MY_PASSWORD >> /tmp/xyz; sleep 365d"]
          resources:
            limits:
              cpu: 200m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 100Mi              

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ekssecrettest
  namespace: dev
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/ekssecrettest

Support for Oracle Cloud Vault

Hi,

I work at Oracle and a customer is using this component, I clone your repo and make an update to support Oracle Cloud Vault

please, check it out and when possible include this directly on your repo.

I don't know Go Lang, so I don't create the files to test the solution and I comment a lot of lint that was on Makefile just to build the container,

The solution is working to Oracle Cloud Vault, but don't have automated test files.

Checkout on my git

https://github.com/herberthnilsen/secrets-init
https://github.com/herberthnilsen/kube-secrets-init

Error: unable to start container process: error during container init

Version 0.4.2
GKE
CloudBuild based Helm deployment.

Here's the error:

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/aece9c11-6a55-443b-8851-1fa6a053f6a4/volumes/kubernetes.io~empty-dir/secrets-init-volume" to rootfs at "/secrets-init/bin": open o_path procfd: open /run/containerd/io.containerd.runtime.v2.task/k8s.io/secrets-init/rootfs/secrets-init/bin: not a directory: unknown

Here's what I'm doing to start the container as one of three other containers inside the pod, the other two depend on this to get at the Google Secrets Manager secrets from the env values specified as they are in your readme, the workload identity service account should be working correctly so I think the issue is specifically mount path related:

- name: secrets-init
  image: ghcr.io/doitintl/secrets-init:latest
  command:
    - sh
  args:
    - -c
    - "cp /usr/local/bin/secrets-init /secrets-init/bin/"
  volumeMounts:
  - mountPath: /secrets-init/bin
    name: secrets-init-volume

I defined the volumes in the yaml as well:

- mountPath: /secrets-init/bin
  name: secrets-init-volume

DoIt Binary Fails in Docker Container When Running Locally (Missing Credentials)

This binary fails when running in a docker container locally. The failure occurs because we are not including any gcp credentials locally, as they are only supplied in production. Can someone augment the code so that there is no segfault if gcp credentials are missing from the container. An example of the docker cmd used is `CMD /usr/local/bin/secrets-init --provider=google .