nemosupremo / vault-gatekeeper

A small service for securely delivering Vault authorization keys to Mesos tasks and ECS containers.

vault-gatekeeper

Vault-Gatekeeper is a small service for delivering Vault token to other services who's lifecycles are managed by a container scheduler such as Mesos or ECS.

Vault-Gatekeeper takes the Cubbyhole Authenication approach outlined by Jeff Mitchell on Vault Blog. Specifically Vault response wrapping is used as outlined in the Vault documentation.

In short, a service will request a vault token from VG supplying its Mesos task id or ECS task arn. VG will then check with Mesos/ECS to ensure that the task has been recently started and that VG has not already issued a token for that task id. Then VG will check its configuration to understand what role that task is assigned and request a response wrapped token from Vault. VG will then pass the token to the service which can then unwrap the response with /sys/wrapping/unwrap to retrieve the token.

Requirements

• Vault 0.6.2+
• Mesos 1.0.0+ (if using Mesos)

Documentation

Visit http://nemosupremo.github.io/vault-gatekeeper

Quickstart

This guide assumes that you 1.) have a Vault instance running, 2.) have a Mesos instance running and 3.) have an approle policy in Vault named test.

1. Install a sample policy in Vault

$ echo '{"mesos:*":{"roles":["test"],"num_uses":1}}' | ./gatekeeper policy update --vault-token 'MY_TOKEN' '-'

2. Start a Gatekeeper instance

$ ./gatekeeper server --mesos-master 'http://leader.mesos:5050' --vault-addr http://localhost:8200

3. Unseal the Gatekeeper instance with a token. (The token must have at least the policy defined in gatekeeper-policy.hcl).

$ ./gatekeeper unseal token --vault-token 'GK_TOKEN'

4. Launch a task on mesos and retrieve a token:

$ curl -X POST -d"{\"task_id\":\"${MESOS_TASK_ID}\"}" 'http://gatekeeper-host/token'

Downloading

You can grab a binary from the releases or deploy the docker image nemosupremo/vault-gatekeeper.

License

MIT

vault-gatekeeper's Issues

Enhancement request: Allow access to token accessor from gatekeeper

When Vault creates a response-wrapped authentication token, the token's accessor is made available in the returned wrap information. This lets privileged callers generate tokens for clients and revoke these tokens (and their created leases) later.

It would be nice to have an API endpoint whereby a privileged caller could supply a task ID and retrieve the token accessor for the token that was generated. This would allow revocation of the token.

Intermittent crash on token request

Hi, we have experienced this issue intermittently where VGM will throw a memory exception when a token is requested (see below).

[31m2016/10/18 19:25:03 [Recovery] panic recovered:
POST /token HTTP/1.1
Host: vgm.foo.com
Accept: */*
Content-Length: 48
Content-Type: application/json
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2


runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:458 (0x43f0f3)
    gopanic: reflectcall(nil, unsafe.Pointer(d.fn), deferArgs(d), uint32(d.siz), uint32(d.siz))
/usr/local/go/src/runtime/panic.go:62 (0x43dc4d)
    panicmem: panic(memoryError)
/usr/local/go/src/runtime/sigpanic_unix.go:24 (0x453af4)
    sigpanic: panicmem()
/go/src/github.com/channelmeter/vault-gatekeeper-mesos/provider.go:63 (0x408295)
    createWrappedToken: defer r.Body.Close()
/go/src/github.com/channelmeter/vault-gatekeeper-mesos/provider.go:107 (0x408b09)
    createTokenPair: return createWrappedToken(token, permTokenOpts, 10*time.Minute)
/go/src/github.com/channelmeter/vault-gatekeeper-mesos/provider.go:202 (0x409924)
    Provide: if tempToken, err := createTokenPair(token, policy); err == nil {
/go/src/github.com/gin-gonic/gin/context.go:97 (0x4c44ba)
    (*Context).Next: c.handlers[c.index](c)
/go/src/github.com/gin-gonic/gin/recovery.go:45 (0x4d3efa)
    RecoveryWithWriter.func1: c.Next()
/go/src/github.com/gin-gonic/gin/context.go:97 (0x4c44ba)
    (*Context).Next: c.handlers[c.index](c)
/go/src/github.com/gin-gonic/gin/logger.go:63 (0x4d3067)
    LoggerWithWriter.func1: c.Next()
/go/src/github.com/gin-gonic/gin/context.go:97 (0x4c44ba)
    (*Context).Next: c.handlers[c.index](c)
/go/src/github.com/gin-gonic/gin/gin.go:284 (0x4ca5be)
    (*Engine).handleHTTPRequest: context.Next()
/go/src/github.com/gin-gonic/gin/gin.go:265 (0x4c9ea0)
    (*Engine).ServeHTTP: engine.handleHTTPRequest(c)
/usr/local/go/src/net/http/server.go:2202 (0x51f2cd)
    serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
/usr/local/go/src/net/http/server.go:1579 (0x51bc37)
    (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
/usr/local/go/src/runtime/asm_amd64.s:2086 (0x46e8d1)
    goexit: BYTE    $0x90    // NOP
�[0m

The chronos task failed, then re-ran. When it re-ran, the token worked fine. Any idea what would cause this? Could we need to allocate more memory to the VGM process?

Thanks
Matt

Enhancement request: Optionally load policies from sub directories

We have encountered a data loss problem when multiple sources update the policies since they are all kept within the single file at /secrets/gatekeeper. Our use case has a large number of policies that support numerous applications where each application team maintains their own policies. One solution is to have each team organize and maintain their policies in named sub-directories. We are working on an optional way to load policies from all sub directories starting with and including /gatekeeper. This load option will be controlled by a new configuration parameter that defaults to false so no changes are required unless someone wants to use this feature.

Unseal failed with the 'approle' method.

gatekeeper unseal approle --auth-app-role $ROLE_ID --auth-app-secret $SECRET_ID
Unsealing gatekeeper at http://localhost:9201
Error from gatekeeper: Unseal failed with the 'approle' method.

approle in vault-
vault write auth/approle/role/gatekeeper
secret_id_ttl=10m
secret_id_num_uses=1
policies=gatekeeper
token_ttl=60m
period=60m

--vault-kv-version not applied

What Im doing. Im run my gatekeeper server with --vault-kv-version 1 flag. But I've get

{"unsealed":true,"error":"The kv backend reported an invalid version. Ensure your kv backend is on version 2."}

just until I'd enable v2 in vault. I need v1 cause spring cloud vault v1.5 doesn't support v2 and I could not use spring boot 2 yet.

Thats my docker run:

docker run --rm -p 9201:9201 -m 128m --cpu-shares=2 --memory-swappiness 0 --name gatekeeper -v /opt/gatekeeper:/opt/gatekeeper nemosupremo/vault-gatekeeper:v1.0.1 server --usage-store vault --schedulers mesos --vault-addr http://test:8200 --mesos-master http://test:5050 --vault-kv-version 1

Thanks

Enhancement request: Support for AppRole Auth Method

Please add support for AppRole Auth Method.

Question on building/installing gatekeeper-cli

I've been following the documentation for how to get this setup ran into a roadblock doing something that seemed like it should be basic.

We must then write our policy to the policy-path where Gatekeeper can expect to find it. The easiest way to do this is with the Gatekeeper Cli.

$ VAULT_TOKEN=some_token gatekeeper policy update --vault-addr http://localhost:8200 ./my-policy.json

This is the first time Gatekeeper Cli is suggested as a tool in the documentation. I can find it's documentation however this documentation doesn't cover anything about how this tool is build/installed. I'm somewhat familiar with Go so I know that generally you can make a Go program run by doing go get -d . to grab the dependencies and then some variant on go build to make the executable but that doesn't appear to work here and even if it did I'm pretty sure that the command I'd need to run it in the end wouldn't start with VAULT_TOKEN=some_token (not even sure why this is in front given that there is a flag for your vault token mentioned in the documentation).

If you're going to suggest we use this tool, could we please have some instructions on how to get/build it.

Add tests

Fix for renewable configurable policy addition with nested policies

PR #54 was incomplete. It should have contained a change to policy_nested.go. PR #55 corrects that.

It also includes an update to the README for the flag and environment variable.

Support list of vault addresses in "-vault" CLI arg (VAULT_ADDR env var)

Currently, the -vault CLI arg (VAULT_ADDR env var) only allows for one Vault address to be listed.
In an environment where there is a cluster with multiple Vault instances, it would be good for the setting to allow for the set of instances to be listed (similar to the MESOS_MASTER setting).

e.g., VAULT_ADDR=https://:8200,:8200,:8200,:8200,:8200

gatekeeper client doesnt work with response wrapping

The client in gatekeeper/ does not work with response wrapping, the permanent token needs to be unmarssalled from the json data in data.response from cubbyhole contents.

This is easily doable with https://godoc.org/github.com/hashicorp/vault/api#Logical.Unwrap

Enhancement request: Create tokens using roles

Currently VGM creates tokens by making requests to Vault at auth/token/create. This means the VGM token needs to have a policy that gives it access to auth/token/create, and with that it can create tokens for any subset of policies that it has.

This has two issues:

1. The VGM token has a lot of power: It can create another token allowing the creation of tokens.
2. Depending on how secrets are organized in Vault, when a new application/service is added, VGM may need to be restarted with a new token that lets it create tokens having access to the new secrets.

This could be more locked down if VGM requested tokens from Vault at auth/token/create/<role_id>. Its token could then have a policy giving it access to auth/token/create/*. This means that the VGM token could create tokens only for named roles (but for any named role).

When new applications were added, a role would be created to allow access to their secrets. Because of the way Vault handles creating tokens with roles, the VGM token would automatically be able to create tokens for the new role; there would be no need to generate a new token for VGM.

Enhancement request: Allow only one request for a token per task ID within startup window

As I understand it, the gatekeeper checks to make sure that a request for a token is coming from a process recently started by Mesos. However, this still leaves open a spoofing attack whereby an attacker uses the task ID to make a second token request successfully.

It would be nice to know that a token can only be requested once for a given task ID within the "recently started" time window. Or is this already implemented?

Unseal fails with self-signed cert

Hello,

It seems vault-gatekeeper fails to unseal correctly if the certificate is self-signed.

vault-gatekeeper unseal approle --auth-app-role $ROLE_ID --auth-app-secret $SECRET_ID --gatekeeper-addr https://localhost:9201
INFO[2019-09-03T22:37:43Z] Unsealing gatekeeper at https://localhost:9201
FATA[2019-09-03T22:37:44Z] Error communicating with gatekeeper: Post https://localhost:9201/unseal: x509: certificate is valid for *.service.consul, not localhost

I didn't see any option to bypass this

Usage:
  gatekeeper unseal [method] [flags]

Flags:
      --gatekeeper-addr string      The address to gatekeeper. (default "http://localhost:9201")
      --vault-token string          Unseal gatekeeper at startup with a Vault token. (default "8809671b-9701-867e-eb29-22a6ac69795d")
      --auth-token-wrapped string   Unseal gatekeeper at startup with a Vault token that is stored with a response wrapped temp token.
      --auth-app-role string        Unseal gatekeeper at startup with a Vault token retrieved using this app role.
      --auth-app-secret string      The app role secret_id to be used.
      --auth-aws-ec2                Unseal gatekeeper at startup using EC2 login.
      --auth-aws-iam string         Unseal gatekeeper at startup using IAM login.
      --auth-aws-nonce string       AWS-EC2 nonce for repeated authentication.
      --auth-gh-token string        Vault authorized github personal token.
  -h, --help                        help for unseal

Is it possible to add a --skip-tls-verify or similar option to unseal?

New Release tag?

Numerous issues and enhancements have been made but the release has not happened for almost one and a half year.

Policy Struct changed

We have been using Vault-Gatekeeper for several years now and I was surprised when I pulled the latest source code to see a change to the policy data struct. It looks like 3 months ago a change was checked in that changed the struct of the policy. In vault we now have to wrap the vault data with another data.

{ "data": { "data": [{"*": { "policies":["default"], "num_uses": 0}]}}

I see the code change in the policy.go file line 173.

mutex issues with TtlSet

It appears that locking is not operating as expected on the TtlSet struct. I was able to replicate this by having multiple forked processed in my application request a token at the same time:

2017/04/26 20:07:10 Provided token pair for x.x.x.x:xxxx in 343.859137ms. (Task Id: task-id-1234) (Task Name: application-name). Policies: [application_policy]
2017/04/26 20:07:10 Provided token pair for x.x.x.x:xxxx in 419.929055ms. (Task Id: task-id-1234) (Task Name: application-name). Policies: [application_policy]                                                                         
2017/04/26 20:07:10 Provided token pair for x.x.x.x:xxxx in 484.607434ms. (Task Id: task-id-1234) (Task Name: application-name). Policies: [application_policy]
2017/04/26 20:07:10 Provided token pair for x.x.x.x:xxxx in 656.629353ms. (Task Id: task-id-1234) (Task Name: application-name). Policies: [application_policy]

Not working with VaultServer

I have a vault in server mode with consul as a backend and I am unable to unseal gatekeeper with vault token

Viewing policies at 'v1/secret/data/gatekeeper'
INFO[2019-07-03T12:01:53+05:30] Attempting to unseal with 'token' method.
INFO[2019-07-03T12:01:53+05:30] Successfully unsealed with 'token' method.
INFO[2019-07-03T12:01:53+05:30] Get Dir List is empty

WARN[2019-07-03T12:01:53+05:30] No policy saved at configured location.

WARN[2019-07-03T12:01:53+05:30] Failed to load policies! Gatekeeper will remain sealed as no policies were loaded. Error: No policy saved at configured location.
FATA[2019-07-03T12:01:53+05:30] Failed to initialize gatekeeper for policy reading: No policies have been configured.

VAULT_TOKEN='token' gatekeeper policy update --vault-addr http://localhost:8200 ./my-policy.json
INFO[2019-07-03T12:14:49+05:30] Reading policy from './my-policy.json'...
INFO[2019-07-03T12:14:49+05:30] Attempting to unseal with 'token' method.
INFO[2019-07-03T12:14:49+05:30] Successfully unsealed with 'token' method.
FATA[2019-07-03T12:14:49+05:30] Failed to write policy:

Only the first AppRole on the roles list is used for authentication

Please correct me if I'm wrong, but

vault-gatekeeper/gatekeeper.go

Line 487 in 4faf5db

roleName := policy.Roles[0]

will always result in using the first item from the roles list, unless the requested role equals task name. Even if there are more AppRoles listed. The loop only checks if requested AppRole is allowed and doesn't overwrite roleName variable. Later it's the roleName not the requestedRoleName that's used in GetRoleId and GetSecretId.

Unit tests also only cover those two cases - single role or role and task name.

So now, if I specify my policy like so:

"roles": [
  "service-sidecar-cache-reader",
  "service-sidecar-cache-writer"
]

and request a token specifying role to be service-sidecar-cache-writer, I'll get a Vault token for the service-sidecar-cache-reader AppRole.

Adding image id for verification of task being launched

I was trying to build something similar but in general instead of verifying from the orchestrator whether a task has been launched with the given name, I was trying to use the image id of the container being launched.

If the two can be combined:

1. If the given service or task has been launched
2. The task has been launched with the given image id

Then it would be more secure to deliver correct secrets to correct containers. Which secrets are to be delivered usually depends on the image and suppose if a task with the same name is launched with different image the secrets can be restricted to be available.

This approach might introduce some more work from the side of the secrets manager because every time an image is updated the image id will change and so the person has to change the vault configuration but this will also prevent leaking secrets into containers that no longer requires it.

Or at least this can be controlled via a flag so that in development mode one can avoid changing the image ids time and again.

Support wildcard inside task name or a more robust (regexp?) matching

Metronome job task names are prefixed with a datetime string, e.g. 20190826135622VHRSz.task-name, so the current matching can effectively only support all tasks at once via mesos:metronome:*. This means that we can't assign a specific AppRole and policy to a specific job "type". Or I don't see how.

Solution: either support an wildcard matching inside a task name, like mesos:metronome:*.task-name or introduce a more robust regexp matchers.

Update to work with Vault 0.6

The documentation (and so, I assume, the codebase) uses the phraseology of Vault <= 0.5 (specifically, "cubbyhole"). Vault 0.6 introduces the concept of "wrapped" tokens, which simplifies the job of this application.

Will vault-gatekeeper-mesos work with the current version of Vault?
When will the code/documentation be updated to the latest version?

typo in policy.go

There is a typo in line 24 (num_users instead of num_uses) which causes the num_uses to be set to 0 despite what you specify in the policy.

{"unsealed":true,"error":"The role requested does not exist."}

I am trying to solve this but not able to get any help.
i have the policy in place as per the documentation.
btw not able to proceed futher because of this.

curl -X POST -d"{"task_id":"n.9bc5088a-9e0f-11e9-a6d9-024258bea194"}" 'http://localhost:9201/token'
{"unsealed":true,"error":"The role requested does not exist."}

New release required for latest security features

Hi @nemosupremo, Please tag the next release as I need AWS_EC2_LOGIN and sys/wrapping/unwrap changes in the Docker Image for added security

Thanks

Mesos 1.8+ no longer supports the state.json endpoint

Testing on the latest release bin from your repo: v1.0.4. I was getting the following generic error:

{"unsealed":true,"error":"json: cannot unmarshal number into Go value of type mesos.mesosState"}

After tcpdumping I noticed it was hitting state.json instead of the state endpoint. I believe state.json was depreciated in mesos 1.8+.

Pluggable Scheduler/Provider backends

While writing tests I came across the fact that we don't only have to check with mesos to see if an incoming request for a token is valid.

If possible we could add:

• A testing backend that fails/grants requests based on some test requirements
• k8s?
• nomad?
• swarm?

Regularly update image via Repository link to pull in security fixes

To get all golang and Alpine security fixes in more frequently an automated build could be set up in Docker Hub listening on upstream repository changes via Repository links.
https://docs.docker.com/docker-hub/builds/#repository-links

If corresponding golang image changes the vault-mesos-gatekeeper image could be rebuilt and re-published under the same tag.

Currently the Alpine base image has several critical security issues that have fixes available.

Task not found response from VGM

I'm using DC/OS 1.9 and Vault 0.7.3. When deploying my container an entrypoint sh script is run that tries to grab the wrapped token from VGM but unfortunately I only get a "task not found" error. If I wait for the deploy to finish and then issue the same command manually from one of the nodes, VGM works as expected returning the wrapped token. Got any ideas on this?

HTTP is enforced to communicate with Mesos

https://github.com/ChannelMeter/vault-gatekeeper-mesos/blob/master/mesos.go#L121

"http://" is hardcoded and it should be possible to use HTTPS as well to avoid eavesdropping.

Token renewal should be retried on failure

Currently a single failure will result gatekeeper sealing it self, even if the error is temporary (network issue, momentarily load etc).
I think that the gatekeeper should perform retries of the token renewals, which would allow it to cope with temporary failures.

Check if slave requesting the token with a given task id, is the same slave running that task id.

We should check (or provide the option to check) that the RemoteAddr that is requesting the token for a given TaskID has the same address as the slave running that TaskID. That way tokens can only be issued to the clients that are running on the same node as that is running the task.

Vault-gatekeeper-mesos will not follow 307 redirect of VAULT_ADDR

Vault handles high availability in active-standby mode. When a user curls a Vault node in Standby mode, it responds with a 307 redirect to the Active vault node's VAULT_ADVERTISE_ADDR:

curl -i -L -H "X-Vault-Token: $VAULT_TOKEN"  -X GET $VAULT_ADDR/v1/secret/?list=true

HTTP/1.1 307 Temporary Redirect
Content-Type: text/plain; charset=utf-8
Date: Wed, 15 Jun 2016 20:36:56 GMT
Location: http://<$VAULT_ADVERTISE_ADDR>/v1/secret/?list=true
Content-Length: 0
Connection: keep-alive

HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 15 Jun 2016 20:36:56 GMT
Content-Length: 147
Connection: keep-alive

I provided VGM with a VAULT_ADDR of our service discovery endpoint for Vault services. VGM doesn't seem to be following the redirects, as I am getting {"status":"Sealed","ok":false,"error":"Error loading policy from vault: 307: communication error."} when it attempts to communicate with the VAULT_ADDR.

I believe that a VGM should inspect a 307 response for the location and then try reaching that, in order to be compatible with a vault HA setup.

Allow client callers to pass config programmatically rather than only via env vars

Currently the client environment variables are loaded in an init() function in the client package, it would be nice for callers to be able to set these themselves by creating a client struct containing the config instead.

There could also be a function to create that struct using env vars for those who wanted to load the config that way.

namespace support?

I know that it is an enterprise feature of vault, but any thought of adding support for namespaces. They alter the paths slightly and are options that can be passed in the CLI or GUI or API.

Replace CubbyUnsealer with WrappedTokenUnsealer

Enhancement request: Support dynamic policy names

Hi,

First of all, thank you for the project.

Idea is to not to maintain the gatekeeper secret/gatekeeper
whenever new micro-service is being created. And make the defaults like this:

   "*": {
        "multi_fetch": true,
        "num_uses": 0,
        "policies": [
            "{{ name }}",
            "default"
        ],
        "ttl": 3000
    }

So based on above example, the gatekeeper, is going to know that service is allowed to use default policy and {{ name }} policy - which is recognized/translated as real name of the service in the end.

This gonna be a killer feature that makes policy handling much easier and will not require policy reloads.

VGM complaining about task running too long

Hi all,

We're seeing an issue in one of our environments where we're seeing occasional failures of VGM to provide a token, with the error message:

2016/10/20 18:30:39 Failed to create token pair for 10.3.1.197:58262 (Task Id: ct:1476988200000:0:legacy_google_verify_addresses:). Reason: This task has been running too long to request a token.

However, according to Mesos, that task had only been running for about a second (those timestamps are the start and stop time, respectively):

ct:1476988200000:0:legacy_google_verify_addresses: ChronosTask:legacy_google_verify_addresses FAILED 2016-10-20T14:30:38-0400 2016-10-20T14:30:39-0400

This is intermittent and happening to both nodes of a two-node VGM installation. VGM and the Chronos tasks are all running in docker containers, and the Docker hosts as well as the Mesos masters are all running NTP and are in sync date-wise.

Any ideas on how to go about further debugging?

Enhancement request: Allow tokens to be non-renewable

I think it would be nice to allow the tokens created by vault-gatekeeper-mesos to be non-renewable upon request. Right now it is hardcoded to true in the create Token pair code and allowing that to be configurable would make this even more flexible.

I am not sure if there is a formal contribution policy or guidelines but I will probably submit a PR for this functionality.

VAULT_TOKEN strategies

This is less of an issue and more of a discussion, but what are some strategies around managing the lifecycle of the credentials provided to vault-gatekeeper-mesos? Using a root token (e.g. the only token which doesn't have an expiration) in VAULT_TOKEN isn't very desirable, so I'm eager to hear your thoughts.

Unclear roles pattern matching

Hi, I spend a lot of time trying to configure gatekeeper properly. Im stuck with role's policies configuration.
This is my policies configuration:

{
        "mesos:*:*": {
                "roles": [
                        "api"
                ],
                "num_uses": 30
        }
}

But when Im trying to get token:

sudo curl -X POST -d"{\"task_id\":\"abm_abm-api.e902c261-94ba-11e8-8961-062024e1b273\", \"scheduler\": \"mesos\", \"role\":\"api\"}" 'http://abmtest:9201/token'

I've got an error instead:

{"unsealed":true,"error":"Your task does not have permission to use this role."}

I have "default" and "api" roles configured in the vault.
When I make my token request without "role" tag:

sudo curl -X POST -d"{\"task_id\":\"abm_abm-api.e902c261-94ba-11e8-8961-062024e1b273\", \"scheduler\": \"mesos\"}" 'http://abmtest:9201/token'

then request goes with "default" role and I could get token.
I've tried "*", "mesos:*:*", "mesos:marathon:*", "mesos:marathon:abm_*" keys but no one works for me. Where Im going wrong and maybe you should clarify documentation or logs output? Thanks!

Travis tests failing

So recently the Travis tests have been failing. The tests use the latest Vault Docker image and with the release of 0.10.0 they introduced versioned secrets. The tests are assuming the old unversioned option of secrets but the secret path of the new image is defaulting to v2.

I will submit a PR shortly that will setup the Vault used by Travis tests to use v1 again. This should make the tests pass again.