gork74 / aws-parameter-bulk

Export AWS SSM Parameter Store values in bulk to .env files

aws-parameter-bulk

Utility to read parameters from AWS Systems Manager (SSM) Parameter Store in bulk and output them in environment-file or json format. It can read all parameters for a given path, or read a list of single parameters. If the parameters contain json, this can be parsed as single values via a flag. It uses your current aws profile to access AWS SSM, you can supply a different profile if you need to read from a different account. Have your AWS CLI set up correctly. See below for instructions.

The output can be used as .env in your development workspace, as --from-env in docker, or as Kubernetes secret.

Install via Homebrew

$ brew tap gork74/gork74

$ brew install aws-parameter-bulk

Usage

get reads names from single values, or from a path recursively. Use --help for usage and parameters.

$ aws-parameter-bulk --help

$ aws-parameter-bulk get --help

Assuming you have the following structure in SSM, and the parameters are filled with "valueOfParam1" etc.:

/dev/test/param1
/dev/test/param2
/dev/test/param3
/dev/other/other1
/dev/other/other2
/dev/testextend/param1
/dev/path/param1
/dev/path/subpath/subparam1
someparam1
someparam2
jsonparam1
jsonparam2

Get Path

These are the outputs you can create for a path variable. Note that the last part of the path will be printed in upper case, if you supply the --upper flag. To be a valid ENV Identifier the output has to use this format: [a-zA-Z_][a-zA-Z0-9_]*

$ aws-parameter-bulk get /dev/test --upper
PARAM1=valueOfParam1
PARAM2=valueOfParam2
PARAM3=valueOfParam3

Get Path without recursion

Paths will be read recursively by default, to turn that off supply the --norecursive flag.

$ aws-parameter-bulk get /dev/path --upper --norecursive
PARAM1=valueOfParam1

Without the flag you would get:

$ aws-parameter-bulk get /dev/path --upper
PARAM1=valueOfParam1
SUBPARAM1=valueOfSubParam1

Get Path with path prefix

If you want to add the full path to the output, use the --prefixpath flag.

$ aws-parameter-bulk get /dev/path --prefixpath
/dev/path/param1=valueOfParam1
/dev/path/subpath/subparam1=valueOfSubParam1

Get Path with normalized path prefix

If you want to add the full path to the output, with underscores as separator (to make it bash variable compliant), use the --prefixnormalizedpath flag. The first slash of the path is not replaced with an underscore, it will be removed.

$ aws-parameter-bulk get /dev/path --prefixnormalizedpath
dev_path_param1=valueOfParam1
dev_path_subpath_subparam1=valueOfSubParam1

Get Multiple Paths

You can supply multiple paths:

$ aws-parameter-bulk get /dev/test,/dev/other --upper
PARAM1=valueOfParam1
PARAM2=valueOfParam2
PARAM3=valueOfParam3
OTHER1=valueOfOther1
OTHER2=valueOfOther2

Overwrite Values

An env file key must be unique, therefore it will be filtered so each key only occurs once. The last key to appear will be printed out, so this will overwrite /dev/test/param1 with /dev/testextend/param1. This can be used to first read some default values and overwrite some of them.

$ aws-parameter-bulk get /dev/test,/dev/testextend --upper
PARAM1=valueOfParamFromExtend1
PARAM2=valueOfParam2
PARAM3=valueOfParam3

JSON Output

Output path parameters as JSON file:

$ aws-parameter-bulk get /dev/test,/dev/other --upper --outjson

{
    "PARAM1": "valueOfParam1",
    "PARAM2": "valueOfParam2",
    "PARAM3": "valueOfParam3",
    "OTHER1": "valueOfOther1",
    "OTHER2": "valueOfOther2"
}

Get Single Parameters

Reading single (non-path) SSM Parameters.

$ aws-parameter-bulk get someparam1,someparam2 --upper
SOMEPARAM1=valueOfSomeParam1
SOMEPARAM2=valueOfSomeParam2

Get Single Parameters on a path

Reading single SSM Parameters on a path.

$ aws-parameter-bulk get /dev/test/param1,/dev/test/param2 --upper
PARAM1=valueOfParam1
PARAM2=valueOfParam2

Get Parameters Containing JSON

Reading SSM Parameters containing JSON, parsing and converting them. This also works for path parameters. Each parameter has to be json.

Assuming this is jsonparam1:

{
  "Json1a": "value1a",
  "Json1b": "value1b"
}

And jsonparam2:

{
  "JSON2a": "value2a",
  "JSON2b": "value2b"
}

This will be the output:

$ aws-parameter-bulk get jsonparam1,jsonparam2 --injson --upper
JSON1A=value1a
JSON1B=value1b
JSON2A=value2a
JSON2B=value2b

Saving From .env File To SSM Names

Takes a file in KEY=value form, and store each line as name and valie in ssm.

$ aws-parameter-bulk save .env
NAME1
NAME2

Saving From .env File To SSM Paths

Takes a file in KEY=value form, prefixes each key with the given path, and stores it in ssm.

$ aws-parameter-bulk save .env /dev/something
/dev/something/PARAM1
/dev/something/PARAM2

Saving From JSON File To SSM Paths

Using a json file as input and storing it to a path

$ aws-parameter-bulk save .env /dev/something --injson

/dev/something/key1=val1
2021-12-07T22:38:19Z INF pkg/util/awsssm.go:174 > Output: {
  Version: 1
}
/dev/something/key2=val2
2021-12-07T22:38:20Z INF pkg/util/awsssm.go:174 > Output: {
  Version: 1
}

Debugging

Add SSM_LOG_LEVEL=debug

$ SSM_LOG_LEVEL=debug aws-parameter-bulk get jsonparam1, jsonparam2 --injson --upper

Web UI

Start with parameter "web" to start a web ui on http://localhost:8888. Change the listen ip and port with the --address flag.

$ aws-parameter-bulk web

$ aws-parameter-bulk web --address :1234

AWS Setup

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html

It is important that you set your region in your aws profile.

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: eu-central-1
Default output format [None]: json

If you have multiple profiles like this (.aws/config):

[default]
account = 11111111111
region = eu-central-1
output = json

[profile other]
account = 2222222222
region = eu-central-1
output = json
source_profile = default

You can read the SSM Parameters from the other account like this:

$ AWS_PROFILE=other aws-parameter-bulk get /dev/test

aws-parameter-bulk's Issues

Feature request: Use underscores in export for prefix path

With a SSM structure like:

/app/foo = foo
/app/bar/foo = foo

and using aws-parameter-bulk get /app --prefixpath --upper --export results in outputting:

export /APP/FOO=foo
export /APP/BAR/FOO=foo

Could this be adapted so it would replace the slashes to underscores instead like so?

export APP_FOO=foo
export APP_BAR_FOO=foo