Auth requests seem nondeterministic about vault-plugin-auth-kubernetes

[jefferai]

Can you please provide logs from the Vault server?

[nateabele]

Wow, you sir are on it. Here you go:

==> Vault server configuration:

             Api Address: http://[::]:8200
                     Cgo: disabled
         Cluster Address: https://172.17.0.8:8201
              Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
                 Storage: inmem
                 Version: Vault v0.10.1
             Version Sha: 756fdc4587350daf1c65b93647b2cc31a6f119cd

WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variable:

    $ export VAULT_ADDR='http://[::]:8200'

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Unseal Key: 3OR5tqUZFn+/U8DB8AuCJn6vQKSXbLxMheLKXyOmDPo=
Root Token: 349ffd4d-31c6-419c-b18a-f6bdb98ce3d8

Development mode should NOT be used in production installations!

==> Vault server started! Log data will stream in below:

2018-07-20T14:44:26.109Z [INFO ] core: security barrier not initialized
2018-07-20T14:44:26.109Z [INFO ] core: security barrier initialized: shares=1 threshold=1
2018-07-20T14:44:26.109Z [INFO ] core: post-unseal setup starting
2018-07-20T14:44:26.123Z [INFO ] core: loaded wrapping token key
2018-07-20T14:44:26.123Z [INFO ] core: successfully setup plugin catalog: plugin-directory=
2018-07-20T14:44:26.123Z [INFO ] core: no mounts; adding default mount table
2018-07-20T14:44:26.126Z [INFO ] core: successfully mounted backend: type=kv path=secret/
2018-07-20T14:44:26.126Z [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2018-07-20T14:44:26.126Z [INFO ] core: successfully mounted backend: type=system path=sys/
2018-07-20T14:44:26.126Z [INFO ] core: successfully mounted backend: type=identity path=identity/
2018-07-20T14:44:26.128Z [INFO ] core: restoring leases
2018-07-20T14:44:26.129Z [INFO ] rollback: starting rollback manager
2018-07-20T14:44:26.130Z [INFO ] identity: entities restored
2018-07-20T14:44:26.130Z [INFO ] identity: groups restored
2018-07-20T14:44:26.130Z [INFO ] core: post-unseal setup complete
2018-07-20T14:44:26.130Z [INFO ] core: root token generated
2018-07-20T14:44:26.130Z [INFO ] core: pre-seal teardown starting
2018-07-20T14:44:26.130Z [INFO ] core: cluster listeners not running
2018-07-20T14:44:26.130Z [INFO ] expiration: lease restore complete
2018-07-20T14:44:26.140Z [INFO ] rollback: stopping rollback manager
2018-07-20T14:44:26.140Z [INFO ] core: pre-seal teardown complete
2018-07-20T14:44:26.140Z [INFO ] core: vault is unsealed
2018-07-20T14:44:26.140Z [INFO ] core: post-unseal setup starting
2018-07-20T14:44:26.140Z [INFO ] core: loaded wrapping token key
2018-07-20T14:44:26.140Z [INFO ] core: successfully setup plugin catalog: plugin-directory=
2018-07-20T14:44:26.140Z [INFO ] core: successfully mounted backend: type=kv path=secret/
2018-07-20T14:44:26.141Z [INFO ] core: successfully mounted backend: type=system path=sys/
2018-07-20T14:44:26.141Z [INFO ] core: successfully mounted backend: type=identity path=identity/
2018-07-20T14:44:26.141Z [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2018-07-20T14:44:26.141Z [INFO ] core: restoring leases
2018-07-20T14:44:26.141Z [INFO ] rollback: starting rollback manager
2018-07-20T14:44:26.141Z [INFO ] identity: entities restored
2018-07-20T14:44:26.141Z [INFO ] identity: groups restored
2018-07-20T14:44:26.141Z [INFO ] core: post-unseal setup complete
2018-07-20T14:44:26.142Z [INFO ] expiration: lease restore complete
2018-07-20T14:44:26.143Z [INFO ] core: mount tuning of options: path=secret/ options=map[version:2]
2018-07-20T14:44:26.146Z [INFO ] secrets.kv.kv_52bc079b: collecting keys to upgrade
2018-07-20T14:44:26.146Z [INFO ] secrets.kv.kv_52bc079b: done collecting keys: num_keys=1
2018-07-20T14:44:26.146Z [INFO ] secrets.kv.kv_52bc079b: upgrading keys finished
2018-07-20T14:48:54.525Z [INFO ] core: successful mount: path=database/ type=database
2018-07-20T14:51:21.553Z [INFO ] core: enabled credential backend: path=kubernetes/ type=kubernetes
2018-07-20T15:15:13.883Z [INFO ] expiration: revoked lease: lease_id=auth/kubernetes/login/f577bf44d775fda31a5c9e7232189ca6aba890df
2018-07-20T15:15:24.950Z [INFO ] expiration: revoked lease: lease_id=database/creds/postgres-role/48ad2786-ecf1-0d7a-bb7b-929b13ccb5ac
2018-07-20T15:15:32.878Z [INFO ] expiration: revoked lease: lease_id=auth/kubernetes/login/fb56aa81190919e8deefe29c22605de5cc5ece78
2018-07-20T16:56:14.442Z [INFO ] expiration: revoked lease: lease_id=database/creds/postgres-role/8e96a82c-e681-a564-2b71-c93ddde74e39

[nateabele]

@jefferai Let me know if I should tune any configs to get more verbose output. Thanks again!

[nateabele]

Hey, checking in. Anything I can do to help move this forward? Thanks.

[nateabele]

I had similar issues with requests sometimes working and sometimes failing in a couple of other contexts (i.e. normal secret lookups with a valid token), and it turned out to be a problem with the default configuration of the Helm Vault Chart, which specifies 3 replicas. If you override this and set it to 1, everything works consistently.

Either the replicas aren't staying in sync (they're backed by a single Consul pod), or not all are communicating with Consul correctly. So, either it's a Vault issue, a Kube issue, or some issue with the stock configuration that deals with how they communicate. 🤷‍♂️

[jefferai]

So are each of these replicas using the same Consul data store? If so it'd be good to know if all but one were showing as standby.

[nateabele]

So are each of these replicas using the same Consul data store?

Yes.

If so it'd be good to know if all but one were showing as standby.

I'll replicate the setup on Monday and find out. Would my issues be expected behavior if the secondary pods were sealed?

[jefferai]

When they're all unsealed, one single Vault instance should show as active and the rest should show as standby.