Describe the issue
A clear and concise description of what the issue is.
Found an issue with the Kubernetes service account role creation in the file
vault-sidecar-injector/deploy/vault/init-dev-vault-server.sh
Lines 33,34,35 bound_service_account_names=default,job-sa
should also include the “vault” service account created by the Vault Helm chart.
Before I modified the locally I could not login using Vault Kubernetes Auth using the test role.
After I modified this locally I could successfully login using Vault Kubernetes Auth but the sidecar injector failed to login.
vault read auth/kubernetes/role/test
Key Value
bound_service_account_names [vault default job-sa]
bound_service_account_namespaces [default]
policies [test_pol]
token_bound_cidrs []
token_explicit_max_ttl 0s
token_max_ttl 0s
token_no_default_policy false
token_num_uses 0
token_period 0s
token_policies [test_pol]
token_ttl 1h
token_type default
ttl 1h
To Reproduce
Steps to reproduce the behavior.
Use the Helm installation instructions here:
https://github.com/Talend/vault-sidecar-injector/blob/master/doc/Deploy.md
git clone https://github.com/hashicorp/vault-helm.git
cd vault-helm
git checkout v0.9.1
helm install vault . --set injector.enabled=false --set server.dev.enabled=true --set ui.enabled=true --set ui.serviceType="NodePort"
Check status
kubectl exec -it vault-0 -- vault status
kubectl logs vault-0
Then init Vault server with our test config:
Set up needed auth methods, secrets engines, policies, roles and secrets
cd vault-sidecar-injector/deploy/vault
./init-dev-vault-server.sh
helm repo add talend https://talend.github.io/helm-charts-public/stable
helm repo update
export CHART_LOCATION=talend/vault-sidecar-injector
helm install vault-sidecar-injector $CHART_LOCATION --namespace default --set vault.addr=http://127.0.0.1:8200 --set vault.ssl.verify=false
kubectl apply -f app-dep-1-secrets.yaml
Expected behavior
A clear and concise description of what you expected to happen.
Expected the secrets to be injected into the app-container but no secrets were injected, as the logs show:
My secrets are:
cat: can't open '/opt/talend/secrets/secrets.properties': No such file or directory
The sidecar injector logs show Vault Kubernetes Auth errors :
{"@Level":"info","@message":"authenticating","@module":"auth.handler","@timestamp":"2021-04-12T03:03:20.730121Z"}
{"@Level":"error","@message":"error authenticating","@module":"auth.handler","@timestamp":"2021-04-12T03:03:20.731143Z","backoff":2.257653078,"error":"Put "http://127.0.0.1:8200/v1/auth/kubernetes/login\": dial tcp 127.0.0.1:8200: connect: connection refused"}
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
Additional context
Add any other context about the problem here.
Content of deployed Kubernetes manifest
If possible, full content of your manifest with Vault Sidecar Injector annotations and info about your workload such as image path, command, env, volumes & mounts ...
The only modification to the original code was to add the vault service account to the initialization script.
Used files from the samples directory without modification.
Logs of Vault Sidecar Injector pod(s)
kubectl logs vault-sidecar-injector-vault-sidecar-inj-65d48c5596-cbqkd
I0412 03:00:54.749279 1 load.go:27] annotationKeyPrefix=sidecar.vault.talend.org
I0412 03:00:54.749476 1 load.go:28] appLabelKey=com.talend.application
I0412 03:00:54.749499 1 load.go:29] appServiceLabelKey=com.talend.service
I0412 03:00:54.749648 1 load.go:103] Loading /opt/talend/webhook/config/injectionconfig.yaml [sha256sum: 967dd69b69017a67bb9ec9e683bb43123618e8e433d5ad05301928b2753138cb]
I0412 03:00:54.751560 1 load.go:103] Loading /opt/talend/webhook/config/proxyconfig.hcl [sha256sum: 65974d444e279560d92773f38188e8bdd09fb3abd19942ed063c2310bd050eb8]
I0412 03:00:54.751638 1 load.go:103] Loading /opt/talend/webhook/config/templateblock.hcl [sha256sum: 9cf401da67775277e99211608f7b26213fb868d1110986043fd20f740894a25d]
I0412 03:00:54.751668 1 load.go:103] Loading /opt/talend/webhook/config/templatedefault.tmpl [sha256sum: be27ce062f7654e64062ffec769f031884676d6d98f1e7c701f601fa42f6d1e1]
I0412 03:00:54.751702 1 load.go:103] Loading /opt/talend/webhook/config/podlifecyclehooks.yaml [sha256sum: a9977154d1ecb2110b63585ee75263aa091e9b3480072f830f58f1ce6ade6a4f]
I0412 03:03:01.780458 1 webhook-server.go:120] AdmissionReview for GroupVersionKind=/v1, Kind=Pod, Namespace=default Name= (app-55b99b4f6f-) UID=632e9899-c4e4-4751-80bd-b439d4e62913 patchOperation=CREATE UserInfo={Username:system:serviceaccount:kube-system:replicaset-controller UID:dfc8504a-1e5e-430d-860b-cff6b52a301b Groups:[system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] Extra:map[]}
I0412 03:03:01.780520 1 utils.go:87] Mutation policy for default/app-55b99b4f6f-: status: "" required:true
I0412 03:03:01.780556 1 update-pod.go:85] Modes status: map[job:false proxy:false secrets:true]
I0412 03:03:01.780586 1 update-pod.go:319] Injecting volumeMount 'secrets' in container 'app-container'
I0412 03:03:01.780606 1 update-pod.go:364] Injecting volume 'secrets' in submitted pod
I0412 03:03:01.780628 1 secrets-func-inject.go:37] [secrets] Injecting container tvsi-vault-agent (path: /spec/containers)
I0412 03:03:01.780890 1 webhook-server.go:142] AdmissionResponse: patch=[{"op":"add","path":"/spec/containers/0/volumeMounts/-","value":{"name":"secrets","mountPath":"/opt/talend/secrets"}},{"op":"add","path":"/spec/volumes/-","value":{"name":"tvsi-shared","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"secrets","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/containers/0","value":{"name":"tvsi-vault-agent","image":"vault:1.6.2","command":["sh","-c","if [ "${VSI_VAULT_AUTH_METHOD}" = "kubernetes" ]; then\n cat \u003c\u003cEOF \u003e vault-agent-config.hcl\npid_file = "/home/vault/pidfile"\n\nauto_auth {\n method "kubernetes" {\n mount_path = "auth/kubernetes"\n config = {\n role = "${VSI_VAULT_ROLE}"\n token_path = "/var/run/secrets/talend/vault-sidecar-injector/serviceaccount/token"\n }\n }\n\n sink "file" {\n config = {\n path = "/home/vault/.vault-token"\n }\n }\n}\n\n${VSI_PROXY_CONFIG_PLACEHOLDER}\n\n${VSI_SECRETS_TEMPLATES_PLACEHOLDER}\nEOF\nelif [ "${VSI_VAULT_AUTH_METHOD}" = "approle" ]; then\n cat \u003c\u003cEOF \u003e vault-agent-config.hcl\npid_file = "/home/vault/pidfile"\n\nauto_auth {\n method "approle" {\n mount_path = "auth/approle"\n config = {\n role_id_file_path = "/opt/talend/secrets/approle_roleid"\n secret_id_file_path = "/opt/talend/secrets/approle_secretid"\n remove_secret_id_file_after_reading = false\n }\n }\n\n sink "file" {\n config = {\n path = "/home/vault/.vault-token"\n }\n }\n}\n\n${VSI_PROXY_CONFIG_PLACEHOLDER}\n\n${VSI_SECRETS_TEMPLATES_PLACEHOLDER}\nEOF\nfi\nif [ "${VSI_JOB_WORKLOAD}" = "true" ]; then\n docker-entrypoint.sh agent -config=vault-agent-config.hcl -tls-skip-verify -log-level=info \u0026\n while true; do\n if [ -f "/opt/talend/tvsi/vault-sidecars-signal-terminate" ]; then\n echo "=\u003e exit (signal received)"\n export VAULT_TOKEN=$(cat /home/vault/.vault-token);\n vault token revoke -tls-skip-verify -self;\n exit 0\n fi\n sleep 2\n done\nelse\n docker-entrypoint.sh agent -config=vault-agent-config.hcl -tls-skip-verify -log-level=info\nfi\n"],"env":[{"name":"SKIP_SETCAP","value":"true"},{"name":"VAULT_ADDR","value":"http://127.0.0.1:8200"},{"name":"VAULT_LOG_FORMAT","value":"json"},{"name":"VSI_JOB_WORKLOAD","value":"false"},{"name":"VSI_PROXY_CONFIG_PLACEHOLDER"},{"name":"VSI_SECRETS_TEMPLATES_PLACEHOLDER","value":"template {\n destination = "/opt/talend/secrets/secrets.properties"\n contents = \u003c\u003cEOH\n {{ with secret "secret/test/test-app-svc" }}{{ range $k, $v := .Data }}\n{{ $k }}={{ $v }}\n{{ end }}{{ end }}\n EOH\n command = ""\n wait {\n min = "1s"\n max = "2s"\n }\n}\n"},{"name":"VSI_VAULT_AUTH_METHOD","value":"kubernetes"},{"name":"VSI_VAULT_ROLE","value":"test"}],"resources":{"limits":{"cpu":"50m","memory":"50Mi"},"requests":{"cpu":"40m","memory":"35Mi"}},"volumeMounts":[{"name":"tvsi-shared","mountPath":"/opt/talend/tvsi"},{"name":"secrets","mountPath":"/opt/talend/secrets"},{"name":"vault-token-47l9p","readOnly":true,"mountPath":"/var/run/secrets/talend/vault-sidecar-injector/serviceaccount"}],"lifecycle":{"preStop":{"exec":{"command":["sh","-c","export VAULT_TOKEN=$(cat /home/vault/.vault-token); vault token revoke -tls-skip-verify -self;\n"]}}},"imagePullPolicy":"Always"}},{"op":"add","path":"/metadata/annotations","value":{"sidecar.vault.talend.org/status":"injected"}}]
I0412 03:03:01.781338 1 webhook-server.go:201] Ready to write reponse ...
kubectl logs vault-sidecar-injector-vault-sidecar-inj-65d48c5596-fchlr
I0412 03:00:41.254234 1 load.go:27] annotationKeyPrefix=sidecar.vault.talend.org
I0412 03:00:41.254328 1 load.go:28] appLabelKey=com.talend.application
I0412 03:00:41.254348 1 load.go:29] appServiceLabelKey=com.talend.service
I0412 03:00:41.254427 1 load.go:103] Loading /opt/talend/webhook/config/injectionconfig.yaml [sha256sum: 967dd69b69017a67bb9ec9e683bb43123618e8e433d5ad05301928b2753138cb]
I0412 03:00:41.256191 1 load.go:103] Loading /opt/talend/webhook/config/proxyconfig.hcl [sha256sum: 65974d444e279560d92773f38188e8bdd09fb3abd19942ed063c2310bd050eb8]
I0412 03:00:41.256253 1 load.go:103] Loading /opt/talend/webhook/config/templateblock.hcl [sha256sum: 9cf401da67775277e99211608f7b26213fb868d1110986043fd20f740894a25d]
I0412 03:00:41.256286 1 load.go:103] Loading /opt/talend/webhook/config/templatedefault.tmpl [sha256sum: be27ce062f7654e64062ffec769f031884676d6d98f1e7c701f601fa42f6d1e1]
I0412 03:00:41.256416 1 load.go:103] Loading /opt/talend/webhook/config/podlifecyclehooks.yaml [sha256sum: a9977154d1ecb2110b63585ee75263aa091e9b3480072f830f58f1ce6ade6a4f]
kubectl logs vault-sidecar-injector-vault-sidecar-inj-65d48c5596-vhddv
I0412 03:00:54.922982 1 load.go:27] annotationKeyPrefix=sidecar.vault.talend.org
I0412 03:00:54.923171 1 load.go:28] appLabelKey=com.talend.application
I0412 03:00:54.923208 1 load.go:29] appServiceLabelKey=com.talend.service
I0412 03:00:54.923338 1 load.go:103] Loading /opt/talend/webhook/config/injectionconfig.yaml [sha256sum: 967dd69b69017a67bb9ec9e683bb43123618e8e433d5ad05301928b2753138cb]
I0412 03:00:54.930367 1 load.go:103] Loading /opt/talend/webhook/config/proxyconfig.hcl [sha256sum: 65974d444e279560d92773f38188e8bdd09fb3abd19942ed063c2310bd050eb8]
I0412 03:00:54.930425 1 load.go:103] Loading /opt/talend/webhook/config/templateblock.hcl [sha256sum: 9cf401da67775277e99211608f7b26213fb868d1110986043fd20f740894a25d]
I0412 03:00:54.930457 1 load.go:103] Loading /opt/talend/webhook/config/templatedefault.tmpl [sha256sum: be27ce062f7654e64062ffec769f031884676d6d98f1e7c701f601fa42f6d1e1]
I0412 03:00:54.930944 1 load.go:103] Loading /opt/talend/webhook/config/podlifecyclehooks.yaml [sha256sum: a9977154d1ecb2110b63585ee75263aa091e9b3480072f830f58f1ce6ade6a4f]
Errors reported by Vault Sidecar Injector while trying to inject Vault Agent into your workload
kubectl logs app-55b99b4f6f-pqff9 tvsi-vault-agent
==> Vault agent started! Log data will stream in below:
==> Vault agent configuration:
Cgo: disabled
Log Level: info
Version: Vault v1.6.2
Version Sha: be65a227ef2e80f8588b3b13584b5c0d9238c1d7
{"@Level":"info","@message":"creating file sink","@module":"sink.file","@timestamp":"2021-04-12T03:03:12.651295Z"}
{"@Level":"info","@message":"file sink configured","@module":"sink.file","@timestamp":"2021-04-12T03:03:12.653247Z","mode":416,"path":"/home/vault/.vault-token"}
{"@Level":"info","@message":"starting template server","@module":"template.server","@timestamp":"2021-04-12T03:03:12.747227Z"}
{"@Level":"info","@message":"starting auth handler","@module":"auth.handler","@timestamp":"2021-04-12T03:03:12.748360Z"}
2021/04/12 03:03:12.748864 [INFO] (runner) creating new runner (dry: false, once: false)
{"@Level":"info","@message":"authenticating","@module":"auth.handler","@timestamp":"2021-04-12T03:03:12.748954Z"}
{"@Level":"info","@message":"starting sink server","@module":"sink.server","@timestamp":"2021-04-12T03:03:12.748510Z"}
2021/04/12 03:03:12.752283 [INFO] (runner) creating watcher
{"@Level":"error","@message":"error authenticating","@module":"auth.handler","@timestamp":"2021-04-12T03:03:12.847056Z","backoff":2.872433287,"error":"Put "http://127.0.0.1:8200/v1/auth/kubernetes/login\": dial tcp 127.0.0.1:8200: connect: connection refused"}
Logs of your workload
My secrets are:
cat: can't open '/opt/talend/secrets/secrets.properties': No such file or directory
cat: can't open '/opt/talend/secrets/secrets.properties': No such file or directory
In particular, logs from the Vault Agent sidecar container injected into your pod