segmentio / chamber

CLI for managing secrets

Chamber

Chamber is a tool for managing secrets. Currently it does so by storing secrets in SSM Parameter Store, an AWS service for storing secrets.

For detailed info about using chamber, please read The Right Way To Manage Secrets

v2.0 Breaking Changes

Starting with version 2.0, chamber uses parameter store's path based API by default. Chamber pre-2.0 supported this API using the CHAMBER_USE_PATHS environment variable. The paths based API has performance benefits and is the recommended best practice by AWS.

As a side effect of this change, if you didn't use path based secrets before 2.0, you will need to set CHAMBER_NO_PATHS to enable the old behavior. This option is deprecated, and We recommend only using this setting for supporting existing applications.

To migrate to the new format, you can take advantage of the export and import commands. For example, if you wanted to convert secrets for service foo to the new format using chamber 2.0, you can do:

CHAMBER_NO_PATHS=1 chamber export foo | chamber import foo -

v2.13.0 Breaking Changes

Support for very old versions of Go has been dropped, and chamber will only test against versions of Go covered by the Go Release Policy, e.g. the two most recent major versions. This will ensure that we can reliably update dependencies as needed. Additionally, chamber binaries will be built with the latest stable version of Go at the time of release.

Installing

If you have a functional go environment, you can install with:

go install github.com/segmentio/chamber/v2@latest

Caveat About chamber version and go install

Note that installing with go install will not produce an executable containing any versioning information. This information is passed at compilation time when the Makefile is used for compilation. Without this information, chamber version outputs the following:

$ chamber version
chamber dev

See the wiki for more installation options like Docker images, Linux packages, and precompiled binaries.

Authenticating

Using chamber requires you to be running in an environment with an authenticated AWS user which has the appropriate permission to read/write values to SSM Parameter Store.

This is going to vary based on your organization but chamber needs AWS credentials to run.

One of the easiest ways to do so is by using aws-vault. To adjust these instructions for your needs, examine the env output of Aws-Vault: How It Works and use your organization's secrets tool accordingly with chamber.

An aws-vault Usage Example With Chamber

aws-vault exec prod -- chamber

For this reason, it is recommended that you create an alias in your shell of choice to save yourself some typing, for example (from my .zshrc):

alias chamberprod='aws-vault exec production -- chamber'

Setting Up KMS

Chamber expects to find a KMS key with alias parameter_store_key in the account that you are writing/reading secrets. You can follow the AWS KMS documentation to create your key, and follow this guide to set up your alias.

If you are a Terraform user, you can create your key with the following:

resource "aws_kms_key" "parameter_store" {
  description             = "Parameter store kms master key"
  deletion_window_in_days = 10
  enable_key_rotation     = true
}

resource "aws_kms_alias" "parameter_store_alias" {
  name          = "alias/parameter_store_key"
  target_key_id = "${aws_kms_key.parameter_store.id}"
}

If you'd like to use an alternate KMS key to encrypt your secrets, you can set the environment variable CHAMBER_KMS_KEY_ALIAS. As an example, the following will use your account's default SSM alias: CHAMBER_KMS_KEY_ALIAS=aws/ssm

Usage

Writing Secrets

$ chamber write <service> <key> <value|->

This operation will write a secret into the secret store. If a secret with that key already exists, it will increment the version and store a new value.

If - is provided as the value argument, the value will be read from standard input.

Secret keys are normalized automatically. The - will be _ and the letters will be converted to upper case (for example a secret with key secret_key and secret-key will become SECRET_KEY).

Listing Secrets

$ chamber list service
Key         Version                  LastModified      User
apikey      2                        06-09 17:30:56    daniel-fuentes
other       1                        06-09 17:30:34    daniel-fuentes

Listing secrets should show the key names for a given service, along with other useful metadata including when the secret was last modified, who modified it, and what the current version is.

$ chamber list -e service
Key         Version                  LastModified      User             Value
apikey      2                        06-09 17:30:56    daniel-fuentes   apikeyvalue
other       1                        06-09 17:30:34    daniel-fuentes   othervalue

Listing secrets with expand parameter should show the key names and values for a given service, along with other useful metadata including when the secret was last modified, who modified it, and what the current version is.

Historic view

$ chamber history service key
Event       Version     Date            User
Created     1           06-09 17:30:19  daniel-fuentes
Updated     2           06-09 17:30:56  daniel-fuentes

The history command gives a historical view of a given secret. This view is useful for auditing changes, and can point you toward the user who made the change so it's easier to find out why changes were made.

Exec

$ chamber exec <service...> -- <your executable>

exec populates the environment with the secrets from the specified services and executes the given command. Secret keys are converted to upper case (for example a secret with key secret_key will become SECRET_KEY).

Secrets from services are loaded in the order specified in the command. For example, if you do chamber exec app apptwo -- ... and both apps have a secret named api_key, the api_key from apptwo will be the one set in your environment.

Reading

$ chamber read service key
Key             Value                           Version         LastModified    User
key             secret                          1               06-09 17:30:56  daniel-fuentes

read provides the ability to print out the value of a single secret, as well as the secret's additional metadata. It does not provide the ability to print out multiple secrets in order to discourage accessing extra secret material that is unneeded. Parameter store automatically versions secrets and passing the --version/-v flag to read can print older versions of the secret. Default version (-1) is the latest secret.

Exporting

$ chamber export [--format <format>] [--output-file <file>]  <service...>
{"key":"secret"}

export provides ability to export secrets in various file formats. The following file formats are supported:

• json (default)
• yaml
• java-properties
• csv
• tsv
• dotenv
• tfvars

File is written to standard output by default but you may specify an output file.

Caveat About Environment Variables

chamber can emit environment variables in both dotenv format and exported shell environment variables. As chamber allows creating key names that are themselves not valid shell variable names, secrets emitted in this format will have their keys modified to confirm to POSIX shell environment variable naming rules:

• variable names must begin with a letter or an underscore
  • variable names must not begin with a number
• variable names must only contain letters, numbers, or underscores

Notes About Dotenv Format

As there is no formal dotenv spec, chamber attempts to adhere to compliance with joho/godotenv (which is itself a port of the Ruby library bkeepers/dotenv). The output should be generally cross-compatible with alternative parsers, but without a formal spec compatibility is not guaranteed.

Of note:

• all key names will be sanitized according the the POSIX shell rules above, and cast to uppercase
• all values will be rendered using special characters instead of string literals, e.g. newlines replaced with the character \n, tabstops replaced with the character \t, etc.
  • no whitespace trimming will be performed on any values

Notes About Exported Environment Variables

Alternatively, chamber may be used to set local environment variables directly with the chamber env command. For example,

source <(chamber env service)`
printf "%s" "$SERVICE_VAR"

Note that all secrets printed this way will be prefixed with export, so if sourced inline as in the above example, then any and all secrets will then be available to any process run after sourcing.

the env subcommand supports output formatting in two specific ways:

chamber env -h
Print the secrets from the parameter store in a format to export as environment variables

Usage:
  chamber env <service> [flags]

Flags:
  -p, --preserve-case    preserve variable name case
  -e, --escape-strings   escape special characters in values

As chamber allows creation of keys with mixed case, --preserve-case will ensure that the original key case is preserved. Note that this will not prevent the key name from being sanitized according to the above POSIX shell rules. By default, values will be rendered using string literals, e.g. newlines will be printed as literal newlines, tabstops as literal tabstops. Output may be emitted using escaped special characters instead (identical to chamber export -o dotenv)) by using the flag --escape-strings.

Importing

$ chamber import [--normalize-keys] <service> <filepath>

import provides the ability to import secrets from a json or yaml file (like the kind you get from chamber export).

Note By default, import will not normalize key inputs, meaning that keys will be written to the secrets backend in the format they exist in the source file. In order to normalize keys on import, provide the --normalize-keys flag

When normalizing keys, before write, the key will be be first converted to lowercase to match how chamber write handles keys.

Example: DB_HOST will be converted to db_host.

You can set filepath to - to instead read input from stdin.

Deleting

$ chamber delete [--exact-key] service key

delete provides the ability to remove a secret from chamber permanently, including the secret's additional metadata. There is no way to recover a secret once it has been deleted so care should be taken with this command.

Note By default, delete will normalize any provided keys. To change that behavior, provide the --exact-key flag to attempt to delete the raw provided key.

Example: Given the following setup,

$ chamber list service
Key         Version                  LastModified      User
apikey      2                        06-09 17:30:56    daniel-fuentes
APIKEY      1                        06-09 17:30:34    daniel-fuentes

Calling

$ chamber delete --exact-key service APIKEY

will delete only APIKEY from the service and leave only

$ chamber list service
Key         Version                  LastModified      User
apikey      2                        06-09 17:30:56    daniel-fuentes

Finding

$ chamber find key

find provides the ability to locate which services use the same key names.

$ chamber find value --by-value

Passing --by-value or -v will search the values of all secrets and return the services and keys which match.

Listing Services

$ chamber list-services [<prefix>]

list-services lists the available services. You can provide a prefix to limit the results.

AWS Region

Chamber uses AWS SDK for Go. To use a region other than what is specified in $HOME/.aws/config, set the environment variable "AWS_REGION".

$ AWS_REGION=us-west-2 chamber list service
Key         Version                  LastModified      User
apikey      3                        07-10 09:30:41    daniel-fuentes
other       1                        07-10 09:30:35    daniel-fuentes

Chamber does not currently read the value of "AWS_DEFAULT_REGION". See https://github.com/aws/aws-sdk-go#configuring-aws-region for more details.

If you'd like to use a different region for chamber without changing AWS_REGION, you can use CHAMBER_AWS_REGION to override just for chamber.

Custom SSM Endpoint

If you'd like to use a custom SSM endpoint for chamber, you can use CHAMBER_AWS_SSM_ENDPOINT to override AWS default URL.

S3 Backend (Experimental)

By default, chamber store secrets in AWS Parameter Store. We now also provide an experimental S3 backend for storing secrets in S3 instead.

To configure chamber to use the S3 backend, use chamber -b s3 --backend-s3-bucket=mybucket. Preferably, this bucket should reject uploads that do not set the server side encryption header (see this doc for details how)

This feature is experimental, and not currently meant for production work.

S3 Backend using KMS Key Encryption (Experimental)

This backend is similar to the S3 Backend but uses KMS Key Encryption to encrypt your documents at rest, similar to the SSM Backend which encrypts your secrets at rest. You can read how S3 Encrypts documents with KMS here.

The highlights of SSE-KMS are:

• You can choose to create and manage encryption keys yourself, or you can choose to use your default service key uniquely generated on a customer by service by region level.
• The ETag in the response is not the MD5 of the object data.
• The data keys used to encrypt your data are also encrypted and stored alongside the data they protect.
• Auditable master keys can be created, rotated, and disabled from the AWS KMS console.
• The security controls in AWS KMS can help you meet encryption-related compliance requirements.

Source: Protecting data using server-side encryption with AWS Key Management Service keys (SSE-KMS)

To configure chamber to use the S3 KMS backend, use chamber -b s3-kms --backend-s3-bucket=mybucket --kms-key-alias=alias/keyname. You must also supply an environment variable of the KMS Key Alias to use CHAMBER_KMS_KEY_ALIAS, by default "alias/parameter_store_key" will be used.

Preferably, this bucket should reject uploads that do not set the server side encryption header (see this doc for details how)

When changing secrets between KMS Keys, you must first delete the Chamber secret with the existing KMS Key, then write it again with new KMS Key.

If services contain multiple KMS Keys, chamber list and chamber exec will only show Chamber secrets encrypted with KMS Keys you have access to.

This feature is experimental, and not currently meant for production work.

Null Backend (Experimental)

If it's preferred to not use any backend at all, use chamber -b null. Doing so will forward existing ENV variables as if Chamber is not in between.

This feature is experimental, and not currently meant for production work.

Analytics

chamber includes some usage analytics code which Segment uses internally for tracking usage of internal tools. This analytics code is turned off by default, and can only be enabled via a linker flag at build time, which we do not set for public github releases.

Releasing

To cut a new release, just push a tag named v<semver> where <semver> is a valid semver version. This tag will be used by Github Actions to automatically publish a github release.

---

THE CHAMBER OF SECRETS HAS BEEN OPENED

chamber's Issues

Document IAM JSON necessary to read KMS keys

I'm currently in the process of reverse engineering this, and it would be nice to know exactly what you need to put in the IAM permission JSON to be able to access your KMS keys.

Supporting parameters in hierarchies

Chamber seems to be using the older pre-hierarchical approach. I need to have my secrets organized in hierarchies.

It seems possible to extend the notion of "service" in Chamber, which is currently [[alphanum]_-], to include "/", and then change the internals to use the GetParametersByPath api call.

In fact a path would replace the current notion of Service and Key.

Chamber separates service and key with ".", but when you have hierarchies you really only want to use "/". Thus there needs to be a consideration for backwards compatibility. One option is that when the service contains a "/", then it is a path, and there is no key and no "." concatenation. Thus:

Old:
chamber write service key value => service.key = value
chamber exec ... service => KEY=value

New:
chamber write multi/level/path value => multi/level/path = value # path combines service and key
chamber write service key value => service.key = value # backwards compatible

chamber exec ... multi/level/ => PATH=value
chamber exec ... multi/ => LEVEL_PATH=value

Does this sound right?

Feature Request: Read capability for ease of use

Love this project! This solved a very complex problem with a very simple solution.

One thing I would like is an easy way to just read a parameter.

I know you can do something like: chamber exec foo_bar -- echo $BAR but that seems overly complex instead of doing something like chamber read foo_bar

Chamber clear env except chamber vars before exec

I believe this would be a nice feature to have particularly for debugging.

Seeing the environment passed through from a command line, it can sometimes be hard to see which environment variables are being set by chamber.

AWS Secrets Manager

AWS has just announced Secrets Manager

• blog: https://aws.amazon.com/blogs/aws/aws-secrets-manager-store-distribute-and-rotate-credentials-securely/
• docs: https://docs.aws.amazon.com/secretsmanager/latest/apireference/Welcome.html

This doesn't completely replace chamber though - chamber has the advantage of being a tiny binary that can sit in your docker container.

Do you think Chamber should switch backends from Parameter Store to use the Secrets Manager API directly? Or do you think chamber should stick to Parameter Store, and perhaps a new chamber-equivalent tool should fill this space?

Can I use chamber as a package in my own Go tool instead of installation?

Hi, Chamber is awesome and I use it for our deployments.
I'm building a tool that deploys one of our components and I'd like to read a parameter using chamber. I don't want the executor to be forced to install Chamber.
I thought about importing the chamber package and use the code.
All I need is the read ability to fetch one value.

Any hints/ideas on how to do that and whether that's even possible?

Different env var to select region?

Hi.

Love the project, and I had an idea which I wanted to float.

We're in a position where we deploy a service across multiple regions, and it requires the same secrets to be saved, regardless of region. We have two options at the moment:

1. Manage a key for each region, ensuring data is synchronised between them
2. Manually force the AWS_REGION env var so that they all use the same region to look for the key.

The issue with the second option is that other parts of the system could be making use of AWS_REGION and might actually need that to work as intended.

Another option could be to look for a different var (e.g. CHAMBER_KMS_REGION) just before https://github.com/segmentio/chamber/blob/master/store/ssmstore.go#L42 and use that instead. I'd be interested to hear your thoughts on this idea!

Thanks

Can secrets be updated only once when shared between services?

First of all: Chamber is really cool :)

Now, let's say I have, for example, an API key for some app which I need in service a and service b. I would do this:

$ chamber write a api_key abcdef1234
$ chamber write b api_key abcdef1234

Now, if that API key changes, how can I make it so I only have to update it in one place? If I have many services with many shared secrets, we quickly lose track of where each secret is used.

Docs request: describe relevant AWS limits

I set this up, and the first time we did a quick(ish) roll of one service, we got throttled by AWS. In the log we see ThrottlingException: Rate exceeded, which is thrown from the call to s.svc.DescribeParameters.

The problem is that I can't figure out which limit is being applied in order to ask AWS to raise it, or if I've set up chamber wrong in some way that is causing it to make more calls than it should.

When trying to recreate this on my laptop, I launched 10 parallel versions of while :; do ./chamber exec dev -- env ; done and triggered the throttle after ~20 requests. This seems very low.

Could you describe which AWS limits are relevant to what parts of chamber's execution, and when we should expect to hit each? If you have any idea which limits they're happy to adjust and which we need to work around that would also be valuable.

Have a way to override KMS key alias with a file

We plan on having a separate KMS key per service, so instead of having to pass in CHAMBER_KMS_KEY_ALIAS for every command, it would be nice if there was a .chamber-kms-key-alias (or something similar) file that chamber looks for in addition to the environment variable so that we can check that file into VCS and it's easier for developers to write secrets for different services.

Happy to put up a PR if this sounds good to you!

region not read from ~/.aws/config

It appears that settings in ~/.aws/config will not set the region.

$ cat ~/.aws/config 
[profile czi]
region = us-west-2
$ AWS_PROFILE=czi chamber exec foo -- ls
Error: Failed to list store contents: MissingRegion: could not find region configuration

Improve documentation regarding parameter_store_key

Currently the code defines parameter_store_key to be used in IAM's Encrypted keys.

var (
	// KeyID is the alias for the KMS key used to encrypt/decrypt secrets
	KeyID = "alias/parameter_store_key"
)

However this key doesn't exist by default and it isn't mentioned in the chamber documentation. You get the below error when trying to write secrets without it.

$ aws-vault exec development -- chamber write test Test "such test"
InvalidKeyId: Alias arn:aws:kms:eu-west-2:xxxx:alias/parameter_store_key is not found. (Service: AWSKMS; Status Code: 400; Error Code: NotFoundException; Request ID: xxxx)
	status code: 400, request id: xxxx

This threw me for a little spin because I wasn't sure if the issue lay elsewhere (i.e. aws-vault) or with chamber. I'm happy to make a pull request with an updated README if that's alright?

As an aside, maybe this is something that users should be able to specify incase they wanted to use a different key?

Command line version flag (and possible update check)

It would be nice to have a --version flag to double check which version of chamber you have installed.

A small upgrade to that would be checking the latest release and linking to the download à la Terraform:

$ terraform --version
Terraform v0.11.2

Your version of Terraform is out of date! The latest version
is 0.11.5. You can update by downloading from www.terraform.io/downloads.html

test

testing github integration

feature request: Allow specification of multiple service names

It would be useful to use chamber like this:

chamber exec myservice mygroup

How do u install chamber?

This might be obvious to a go dev, so forgive me. But I see no mention of this throughout this repo, so I thought I'd ask.

Reading versions of secrets greater than v50 but less than current version fails

A chamber read my_service my_secret -v 56 of a secret named my_secret with current version as version 56 will give the current secret, but a chamber read my_service my_secret -v 50<version_number<current of any versions of the secret between v51 and current returns Error: Failed to read: secret not found.

For eg chamber read my_service my_secret -v 55 or chamber read my_service my_secret -v 54 or chamber read my_service my_secret -v 53 or chamber read my_service my_secret -v 52 or or chamber read my_service my_secret -v 51 fails.

A chamber read my_service my_secret -v 50 and version numbers below 50 returns values correctly.

Alpine package?

Hi,

Thanks for this tool - is there any interest in creating an Alpine package for it?

bug: chamber write unable to write value that begin with --

chamber will attempt to treat the value (i.e. an RSA key) into a command line parameter. you can manually create a well formed parameter with the AWS cli so chamber can read it later, but chamber can't import a --- prefixed parameter.

sorry, i don't have any go programming smarts, otherwise i would attempt a patch for this that allows passing in the secret via stdin or somesuch. thanks!

Feature Request: Support AWS Tags

A great feature to see would be the ability to add tags to secrets with command line flags when using the write command. In our organization we use tags to quickly see all resource usage on a project-per-project basis (for projects that share our common account and don't warrant their own account) using AWS System Manager resource groups.

This is possible using the AddTagsToResource Systems Manager API call.

Syntax would be something like:

$ chamber write my_service my_key my_value --tag tag1=value1 --tag tag2=value2

I'll try to learn me some go and see if I can whip up a PR when I find the time but thought I'd document this here and see if there's any interest.

Does chamber support multiple environments in the same region?

Great tool. I discovered chamber via an article on AWS blog.

My question is the following: does chamber support multiple environments in one region?

For historical reasons, we run two different environments in the same region but in different VPC. I'd love to use chamber to manage our secrets but seems like there is no easy way to do so. To support such functionality chamber needs to support some sort of namespaces/prefixes in the parameter store's key name.

Right now the key name format is the following:
service_name/key_name

Does chamber something like:
namespace/service_name/key_name

Thanks in advance!

Intended workflow for docker restart policy always

From what I understand, chamber is either added to the docker image (or used as a sidecar?) with the AWS tokens set as environment variables and the chamber executable is prepended to the entrypoint. As I understand this will rerun chamber every time the container fails, and if I understand AWS correctly, the AWS tokens will expire after some time and then the container needs new tokens, requiring manual intervention.

Is there a better way to automate this situation?

Feature Request: Output shell script to set environment variables

Like using exec for a specific process, something like chamber populate will grab secrets and then evaluate them or export into the current environment.

Feature Request: Don't update value if there is no change

It would be great if chamber doesn't update the value on a key-value pair when the "new" value is the same as the existing. This prevents superfluous items being added to the history either by mistake or automation.

Feature request: export -f dotenv

Hi guys, thanks for your work on chamber and all the open communication you've been publishing with regards to Terraform, AWS, aws-vault, etc. It's very helpful to a lot of people.

Here's a simple feature that I'd like to see: Instead of using environment variables, I would prefer to store the secrets in a .env file. And I'd like to populate that .env file via chamber.

I propose adding a format option to export of dotenv. I envision it working like so:

chamber export service_name -f dotenv

The output would simply be VARIABLE=value rows:

SECRET_KEY=abc
FREE_TACOS=true

etc.

In production, we'd simply run something along the lines of chamber export service_name -f dotenv > config/.env.

This way, we make it a little tougher to accidentally leak secrets through environment debug logs, etc that may crop up from 3rd party executables.

If there's interest in this, I may be able to find the time to implement it myself and send over a PR.

Workarounds for DescribeParameters Rate-Limits

Summary

I've been experimenting with using chamber to load and manage secrets for our micro-service app and been running head-first into AWS throttling for calls to the DescribeParameters API. I'd like to explore ways of mitigating these limits in chamber.

Prior Work

Other people seem to have hit similar problems in the past:

• #34
• #36
• #54

Problem Statement

When we bring up a bunch of new services (eg. when scaling-up or creating a new stack for blue-green deployment), we run chamber exec or chamber export in each of our docker containers to load the secrets for that process. Chamber then calls DescribeParameters to list the available parameter names for the service, which then hangs during exponential backoff in the AWS SDK until it succeeds (for the first few containers) then runs-out of retries and fails for the rest.

We've tried increasing the number of retries, but it just takes too long (which is a big problem if we're bringing-up new containers to scale out-of an increase in load). The AWS rate-limit appears to be very low (less than one per second).

Solutions

Obviously the best solution would be to increase the AWS rate-limits, but they seem very unwilling to do that for DescribeParameters (they insist that it should only be used for manual querying on a CLI). I've created this issue to gauge interest and survey ideas for building some sort of workaround into chamber.

Whatever we build will need to find some alternate way of listing the relevant parameters, then just query for their latest values using GetParameters (which has much higher limits). Here are some ideas I've been thinking about:

• Option 1: Cache parameter names in a file. Dump the parameter names to a file (eg .chamber-<service-name>-cache.json) after successfully running DescribeParameters, then read them from the file next-time (if the caller provides the --use-cache option), with some freshness limit on the modification time of the file (eg. 1 hour). This would work transparently most of the time, but also let users pre-generate and ship-around the cache files as build artefacts if they wanted.
• Option 2: Pass in the parameter names as arguments. This is simple, but ugly. It forces everyone with this problem to come-up with their own scheme for caching and/or shipping-around parameter names, but it also gives maximum flexibility about how to do that.
• Option 3: Cache the full list of names as a parameter-store parameter. When the chamber CLI writes/deletes a parameter, it could fetch the full (updated) list of parameter names and store it in an (unencrypted?) parameter (eg. <service-name>.chamber-internal-parameter-name-cache). Then it could optionally use the value of that parameter as a cache when reading. This is the most magic and transparent solution, but it will be confusing when it breaks and it has some awkward edge-cases (eg. hitting the max parameter value size, race-conditions between chamber instances to update the list, restrictive permissions on IAM roles for reading/writing to the internal parameter).

Maybe there's a better solution I haven't thought-of; I'd love input from anyone else who's had this problem and found a work-around.

Support Hierarchical Keys

I'm still just exploring chamber but I'm confused on how to properly make use of hierarchical keys. Am I just to do underscores to mimic a hierarchy? Example:

chamber write appname_env_region_mt keyfoo valuebar

I believe parameter store has some ability to use / delimiter in keys? Is that not supported? I tried:

→ chamber write /appname/env/region/mt keyfoo valuebar

Failed to validate service name '/appname/env/region/mt'. Only alphanumeric, dashes, and underscores are allowed for service names

We, rather unfortunately, do not have separate AWS accounts for each environment. Still, even if we did, we are multi-region and have both multi-tenant and customer specific clusters. Our need for key hierarchies would remain.

Perhaps I'm just doing it wrong?

Link to the blog post Evan wrote

feature request: `retries` to be picked up when creating ssmSession / getting NoCredentialProviders

aws-sdk-go has relatively short timeout when fetching credentials, which shows up as NoCredentialProviders when using kube2iam.

The credentials are eventually passed through and you can work around by adding a sleep 5 before chamber. But would be nice for chamber to retry (maybe with delay)?

Looking through ssmstore - retries are set for SSM request, but not for the SSM session itself. Would be awesome to have it there, too.

consider calling syscall.Exec

execRun does a ton of work to pass through stdin, stdout, signals, return codes, etc from a single child process.

Instead of doing that I am wondering if it would be easier to simply call syscall.Exec with the arguments you want. That would replace the current process with the child process, and you wouldn't need to worry about signals or inputs/outputs being passed through correctly.

https://golang.org/pkg/syscall/#Exec

securing different enviroments

Is there a best practice on the best way to limit exposure of different env to diff services?

feature request: write config entries as an ENV file

Environment files can be used with systemd services (EnvironmentFile=) or other daemon management tools (foreman, etc).

Can be usefull in some scenarios. An uglish workaround for now is:

env - /usr/bin/chamber exec myapp -- /usr/bin/env > /run/myapp/env

Using different environments in ENTRYPOINT

First of all, thank you for creating this solution. What solution do you use for letting the parameter vary in the entrypoint? e.g. ENTRYPOINT ["chamber", "exec" "someenv", "--"]? This changes for us in each cluster we setup. It's simple to have in an entrypoint but Docker doesn't support passing variables there. The only other solution I can think of is to not have chamber be the entrypoint but have a different init process. thanks.

Document required AWS permissions to run policy

chamber crashes if description of the parameter is not defined

Hi Team,

Chamber fails when at least one parameter within service contains empty description.
Such limitation makes not possible to use terraform to provision parameters,
it doesn't have possibility to set description for ssm parameter.

I see you use "description" for versioning purposes, however, parameter store already enabled versioning out of the box.

Please have a look.

Problem:

$chamber list testenv
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x809048]

goroutine 1 [running]:
github.com/segmentio/chamber/store.parameterMetaToSecretMeta(0xc4201c41c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/segmentio/chamber/store/ssmstore.go:310 +0x48
github.com/segmentio/chamber/store.(*SSMStore).List(0xc420013f40, 0x7fffd19327d3, 0x7, 0x0, 0x40ca5f, 0x8b0f00, 0xc420013afd, 0xc4200d5b87, 0xc420013afd)
/go/src/github.com/segmentio/chamber/store/ssmstore.go:220 +0x80a
github.com/segmentio/chamber/cmd.list(0xc754a0, 0xc420013bd0, 0x1, 0x1, 0x0, 0x0)
/go/src/github.com/segmentio/chamber/cmd/list.go:39 +0x105
github.com/segmentio/chamber/vendor/github.com/spf13/cobra.(*Command).execute(0xc754a0, 0xc420013b80, 0x1, 0x1, 0xc754a0, 0xc420013b80)
/go/src/github.com/segmentio/chamber/vendor/github.com/spf13/cobra/command.go:647 +0x3f8
github.com/segmentio/chamber/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc758e0, 0x4, 0xc75b00, 0xc75a20)
/go/src/github.com/segmentio/chamber/vendor/github.com/spf13/cobra/command.go:726 +0x339
github.com/segmentio/chamber/vendor/github.com/spf13/cobra.(*Command).Execute(0xc758e0, 0xc420056058, 0x0)
/go/src/github.com/segmentio/chamber/vendor/github.com/spf13/cobra/command.go:685 +0x2b
github.com/segmentio/chamber/cmd.Execute()
/go/src/github.com/segmentio/chamber/cmd/root.go:42 +0x31
main.main()
/go/src/github.com/segmentio/chamber/main.go:6 +0x20

Developing & Prod with Chamber

Hello there, I have a question about using chamber, not sure if this is the best place to ask it, please feel free to redirect me somewhere else :)

I have read your article The right way to manage secrets and was just wondering how your developers are using chamber locally with Docker to develop offline. My understanding is that chamber connects to the AWS Parameter Store to fetch credentials (which wouldn't work offline). I see that aws-vault has a local server mode - does this fulfill that purpose somewhat?

My question is twofold:

1. How do you keep the code calling chamber in Dockerfiles (the entrypoint) the same for prod and development? As I understand aws-vault is required to be used locally, while not required in prod due to EC2/ECS IAM roles.

2. Can you use chamber offline whilst starting Docker containers with some sort of dummy Parameter Store? How do your developers accomplish this?

Thanks in advance :)

Feature Request: chamber delete(rm) <service> or <service> <key>

Basically do the opposite of write with the added functionality of deleting an entire service if applicable.

IE: chamber delete service would delete all keys with that namespace
or
chamber delete service key would delete just that specific key

Addition it will have to delete in batches of 10 due to the AWS constraint: Member must have length less than or equal to 10

Warn if SSM API requests are throttled

Under normal operation, the SSM API will respond reasonably quickly to requests.

When SSM API requests are throttled, Chamber currently retries the requests up to the configured retry count. While it does so, no output is returned to the user, and Chamber appears to hang in the meantime.

It would be useful to know when Chamber is retrying the requests so that the user can know that progress is being made. This can help rule out other potential problems such as network connectivity or other issues. Otherwise, the user is only aware of the symptom of the problem (hanging) and has no indication of the cause.

Support writing secret values from stdin

It would be nice to be able to write secrets using stdin, for example if you wanted to take file contents and write that as a secret with cat foo | chamber write service key.

"chamber write:" add option to read from os.Stdin

If I run chamber write service key, I get an error about too few arguments.

Passing a secret value at the command line means it may leak into bash_history or e.g. audit tools that export all run commands to a third party server. This is one reason that Mysql's -p flag prompts you for the password by default when you enter it at the command line.

It would be nice if "chamber write" supported this same functionality.

Feature Request: chamber load <service> <file>

Not sure if it's something you want to tackle or not, but would load the ability to specify an .env file or just a file with key value pairs.

We have a ton of rails apps that we are converting to use chamber and it would be really nice to just do something like:

chamber load someservice .env or something similar.
Bonus points if it can exclude new lines and comments by default!
More bonus points if I don't have to remove all the equal signs in the file!

I can definitely see this solution becoming a lot more popular with native AWS users, the simplicity is great.

Chamber fails to list, read or write secrets when it is a private key

I'm not sure if this is a bug or I'm going it wrong but I can't get chamber to read or write a secret when the value is a private key (-----BEGIN RSA PRIVATE KEY-----) etc... I can store these values in the GUI but chamber can't decrypt it.

chamber read david git_priv_key
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x14015ba]

goroutine 1 [running]:
github.com/segmentio/chamber/store.parameterMetaToSecretMeta(0xc4203482c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/private/tmp/chamber-20171206-23249-ulm2p7/chamber-1.13.0/src/github.com/segmentio/chamber/store/ssmstore.go:404 +0x4a
github.com/segmentio/chamber/store.(*SSMStore).readLatest(0xc4200dcfe0, 0x7fff5fbff6f5, 0x5, 0x7fff5fbff6fb, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/private/tmp/chamber-20171206-23249-ulm2p7/chamber-1.13.0/src/github.com/segmentio/chamber/store/ssmstore.go:243 +0x53e
github.com/segmentio/chamber/store.(*SSMStore).Read(0xc4200dcfe0, 0x7fff5fbff6f5, 0x5, 0x7fff5fbff6fb, 0xc, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
/private/tmp/chamber-20171206-23249-ulm2p7/chamber-1.13.0/src/github.com/segmentio/chamber/store/ssmstore.go:118 +0xda
github.com/segmentio/chamber/cmd.read(0x18b19c0, 0xc4200dce80, 0x2, 0x2, 0x0, 0x0)
/private/tmp/chamber-20171206-23249-ulm2p7/chamber-1.13.0/src/github.com/segmentio/chamber/cmd/read.go:56 +0x183
github.com/segmentio/chamber/vendor/github.com/spf13/cobra.(*Command).execute(0x18b19c0, 0xc4200dce00, 0x2, 0x2, 0x18b19c0, 0xc4200dce00)
/private/tmp/chamber-20171206-23249-ulm2p7/chamber-1.13.0/src/github.com/segmentio/chamber/vendor/github.com/spf13/cobra/command.go:647 +0x3f1
github.com/segmentio/chamber/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x18b1be0, 0x0, 0x3a, 0x18b0f20)
/private/tmp/chamber-20171206-23249-ulm2p7/chamber-1.13.0/src/github.com/segmentio/chamber/vendor/github.com/spf13/cobra/command.go:726 +0x2fe
github.com/segmentio/chamber/vendor/github.com/spf13/cobra.(*Command).Execute(0x18b1be0, 0xc420105f30, 0x1456ade)
/private/tmp/chamber-20171206-23249-ulm2p7/chamber-1.13.0/src/github.com/segmentio/chamber/vendor/github.com/spf13/cobra/command.go:685 +0x2b
github.com/segmentio/chamber/cmd.Execute()
/private/tmp/chamber-20171206-23249-ulm2p7/chamber-1.13.0/src/github.com/segmentio/chamber/cmd/root.go:42 +0x31
main.main()
/private/tmp/chamber-20171206-23249-ulm2p7/chamber-1.13.0/src/github.com/segmentio/chamber/main.go:6 +0x20

use '/' for hierarchy rather than '.'

Would it make sense to use '/' to create a hierarchy rather than using the '.'? This way we could use IAM permissions to create service-level access policies to the parameters.

For example,

chamber write myservice foo 1234

would yield a parameter named: myservice/foo and we could use the following statement for the service:

 {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParametersByPath"
            ],
            "Resource": "arn:aws:ssm:us-east-1:123456789:parameter/myservice/*"
}

Doc: http://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-paramstore-working.html

chamber error: Failed to read: secret not found

I'm seeing weird behavior when trying to read a secret. When I chamber list, it shows up, but when trying to read, I get a "not found" error:

∴ aws-okta exec stage -- chamber list myservice
2018/01/08 11:33:47 Opening keychain /Users/stephenmathieson/Library/Keychains/aws-okta.keychain
Key			Version		LastModified	User
segment_password	1		01-05 15:04:17	arn:aws:sts::355207333203:assumed-role/AdministratorAccess/1515182656081824000
segment_username	1		01-05 15:03:43	arn:aws:sts::355207333203:assumed-role/AdministratorAccess/1515182622231938000

∴ aws-okta exec stage -- chamber read myservice segment_username
2018/01/08 11:34:03 Opening keychain /Users/stephenmathieson/Library/Keychains/aws-okta.keychain
chamber error: Failed to read: secret not found
exit status 1

Any idea what might be happening?

cli displays lowercase even though its stored in uppercase

It's confusing that the env var name is stored in uppercase but displayed in lowercase

Documentation: Setting Region

Fantastic tool, thank you for sharing!

I usually set my region via export AWS_DEFAULT_REGION=us-east-2 in my shell environments. I was therefore confused with the following when I tried to set a secret on my workstation:

$ chamber write testservice testkey testvalue
MissingRegion: could not find region configuration

I found the aws-sdk-go documentation on setting the region and using AWS_REGION=us-east-2 worked for me. I know this is an SDK issue, not chamber's, but perhaps it might be helpful to mention this in the project README; I could submit a PR if that would be welcomed.

Feature Request: Option to Overwrite Envs by Caller

what

• allow user to call chamber exec and not overwrite specific variables

why

• Allow variables in caller's shell to override chamber secrets

use-case

• We have all secrets in chamber, but sometimes we want to override a value in our CI pipeline for testing

example implementation

chamber exec -o DB_USER -o DB_PASS -o DB_HOST -- app

This would let the caller's environment overwrite DB_USER, DB_PASS, DB_HOST

Support for empty values

Might not make a lot of sense, but bear there with me - the use case is for chamber reading keys from the parameter store - and be able to have environment variables with empty values ("").

Parameter Store has this feature request 6 years in the making, so not likely to support. (its for CloudFormation, but root cause seems to be the same).

Feature request to chamber is to work around it by, probably, using some sort of placeholder for empty values.

Feature Request: Rollback secrets version

Since chamber stores on SSM and the secrets gets their versions incremented, I'd like to have the option to revert the keys to a previous version with something like chamber revert <service> <key>

Relevant use case: I've updated a secret / env variable with chamber, then the deployed production service fails to load due to the bad key, I'd like to quickly have the option to revert.
In my case, I'd be doing it as a logical step in my CI deployment process:

1. Sync keys and variables using chamber import
2. Deploy version to production
3. If version fails revert chamber changes

Consider Organizing Parameters into Hierarchies using "/" instead of "."

You are currently using the "." to organize the parameters, it seems that "/" would be a better choice.

CF: https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-paramstore-working.html