codahale / sneaker

A tool for securely storing secrets on S3 using Amazon KMS.

Sneaker

Setec Astronomy? Keynote Shogun.

sneaker is a utility for storing sensitive information on AWS using S3 and the Key Management Service (KMS) to provide durability, confidentiality, and integrity.

Secrets are stored on S3, encrypted with AES-256-GCM and single-use, KMS-generated data keys.

Table Of Contents

• WARNING
• Installing
• Using
  • Configuring Access To AWS
  • Setting Up The Environment
  • Basic Operations
  • Packing Secrets
  • Unpacking Secrets
  • Encryption Contexts
  • Maintenance Operations
• Implementation Details
• Architecture
• Threat Model
  • Assumptions
  • Threats From A KMS Compromise
  • Threats From An S3 Compromise
  • Threats From Seizure Or Compromise Of The User's Computer

WARNING

This project has not been reviewed by security professionals. Its internals, data formats, and interfaces may change at any time in the future without warning.

Installing

go get -d -u github.com/codahale/sneaker
cd $GOPATH/src/github.com/codahale/sneaker
make install
sneaker version

Using

Configuring Access to AWS

sneaker requires access to AWS APIs, which means it needs a set of AWS credentials. It will look for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, the default credentials profile (e.g. ~/.aws/credentials), and finally any instance profile credentials for systems running on EC2 instances.

In general, if the aws command works, sneaker should work as well.

If you have multi-factor authentication enabled for your AWS account (and you should), you may need to provide a token via the AWS_SESSION_TOKEN environment variable.

If you're using IAM instance roles, you may need to set the AWS_REGION environment variable to the AWS region you're using (e.g. us-east-1).

Setting Up The Environment

Sneaker requires two things: a KMS master key and an S3 bucket.

You can create a KMS key via the AWS Console or using a recent version of aws. When you've created the key, store its ID (a UUID) in the SNEAKER_MASTER_KEY environment variable:

export SNEAKER_MASTER_KEY="9ed356fb-5f0f-4792-983d-91866faa3705"

As with the key, you can create an S3 bucket via the AWS Console or with the aws command. You can either use a dedicated bucket or use a directory in a common bucket, but we recommend you do two things:

1. Use a Private ACL. In addition to the cryptographic controls of sneaker, access control is critical in preventing security breaches.

2. Enable access logging, ideally to a tightly-controlled, secure bucket. While Amazon's CloudTrail provides audit logging for the vast majority of AWS services, it does not do so for S3 access.

Once you're done, set the SNEAKER_S3_PATH environment variable to the location where secrets should be stored (e.g. s3://bucket1/secrets/):

export SNEAKER_S3_PATH="s3://bucket1/secrets/"

(That will store the encrypted secrets in the bucket bucket1 prefixed with secrets/.)

Basic Operations

Once you've got sneaker configured, try listing the secrets:

sneaker ls

This will print out a table of all uploaded secrets. You haven't uploaded anything yet, so the table will be empty.

Let's create an example secret file and upload it:

echo "This is a secret!" > secret.txt
sneaker upload secret.txt example/secret.txt

This will use KMS to generate a random, 256-bit data key, encrypt the secret with AES-256-GCM, and upload the encrypted secret and an encrypted copy of the data key to S3. Running sneaker ls should display a table with the file in it.

If your file is so sensitive it shouldn't be stored on disk, using - instead of a filename will make sneaker read the data from STDIN.

You can download a single file:

sneaker download example/secret.txt secret.txt

Finally, you can delete the file:

sneaker rm example/secret.txt

Packing Secrets

To install a secret on a machine, you'll need to pack them into a tarball:

sneaker pack example/*,extra.txt example.tar.enc

This will perform the following steps:

1. Download and decrypt all secrets matching any of the patterns: example/* or extra.txt.

2. Package all the decrypted secrets into a TAR file in memory.

3. Generate a new data key using KMS.

4. Use the data key to encrypt the TAR file with AES-GCM.

5. Write both the encrypted data key and the encrypted TAR file to example.tar.enc.

Using - as the output path will make sneaker write the data to STDOUT.

The result is safe to store and transmit -- only those with access to the Decrypt operation of the KMS key being used will be able to decrypt the data.

You can also use a different KMS key than your SNEAKER_MASTER_KEY when packing secrets by using the --key-id flag:

sneaker pack example/* example.tar.enc --key-id=deb207cd-d3a7-4777-aca0-01fbceb4c927

This allows you to unpack your secrets in environments which don't have access to the key used to store your secrets.

Unpacking Secrets

To unpack the secrets, run the following:

sneaker unpack example.tar.enc example.tar

This will perform the following steps:

1. Read example.tar.enc.

2. Extract the encrypted data key and encrypted TAR file.

3. Use KMS to decrypt the data key.

4. Decrypt the TAR file and write the result to example.tar.

Using - instead of a filename will make sneaker read the data from STDIN. Likewise, using - as the output path will make sneaker write the data to STDOUT. This allows you to pipe the output directly to a tar process, for example.

Encryption Contexts

KMS supports the notion of an Encryption Context: semi-structured data used in the encryption of data which is then required for resulting decryption operations to be successful.

sneaker uses the SNEAKER_MASTER_CONTEXT environment variable as the default encryption context for the secrets which are stored in S3. In addition, sneaker also includes the full S3 path, including bucket and prefix. Because of this, secrets in S3 cannot be renamed; they can only be deleted and re-uploaded.

Note: there is currently no way to change the contents of SNEAKER_MASTER_CONTEXT in place. If you want to change it, you'll need to download all your secrets and re-upload them with the new context.

For packing and unpacking secrets you can specify a different encryption context on the command line:

sneaker pack example/* secrets.tar.enc --context="hostname=web1.example.com,version=20"

That same context (hostname=web1.example.com,version=20) must be used to unpack those secrets:

sneaker unpack secrets.tar.enc secrets.tar --context="hostname=web1.example.com,version=20"

All data in the encryption contexts are logged via CloudTrail, which allows you to track when and where particular secrets are packed and unpacked.

Maintenance Operations

A common maintenance task is key rotation. To rotate the data keys used to encrypt the secrets, run sneaker rotate. It will download and decrypt each secret, generate a new data key, and upload a re-encrypted copy.

To rotate the KMS key used for each secret, simply specify a different SNEAKER_MASTER_KEY and run sneaker rotate.

Implementation Details

All data is encrypted with AES-256-GCM using random KMS data keys and random nonces. The ID of the KMS key is used as authenticated data.

The final result is the concatentation of the following:

• A four-byte header of the length of the encrypted KMS data key, in bytes, in network order.

• The encrypted KMS data key, verbatim. (This is an opaque Amazon format which includes the key ID.)

• The AES-256-GCM ciphertext and tag of the secret.

Architecture

Threat Model

The threat model is defined in terms of what each possible attacker can achieve. The list is intended to be exhaustive, i.e. if an entity can do something that is not listed here then that should count as a break of Sneaker.

In broad strokes, the confidentiality and integrity of content stored with Sneaker is predicated on the integrity of Sneaker, the confidentiality and integrity of KMS, and the integrity of S3.

Assumptions

• The user must act reasonably and in their best interest. They must not reveal secrets or allow unauthorized access to secrets.

• The user must run a copy of Sneaker which has not been suborned.

• The user's computer must function correctly and not be compromised by malware.

• Communications with Amazon have confidentiality and integrity ensured by the use of TLS.

• Key Management Service is reasonably secure: its keys will not be compromised, its random numbers are unguessable to adversaries, its cryptographic algorithms are correct.

• The authentication and access control functionality of both KMS and S3 are secure.

• AES-256 and GCM's security guarantees are valid.

Threats From A KMS Compromise

An attacker who suborns KMS can:

• Create forged secret packages.
• Decrypt packaged tarballs.
• Deny the ability to decrypt secrets, either temporarily or permanently.

Threats From An S3 Compromise:

An attacker who suborns S3 can:

• Delete or modify secrets such that they are no longer valid.

Threats From Seizure Or Compromise Of The User's Computer

An attacker who physically seizes the user's computer (or compromises the user's backups) or otherwise compromises it can:

• Recover AWS credentials and pose as the user. If multi-factor authentication is not enabled, this would allow the attacker to extract all secrets, modify secrets, etc.

sneaker's Issues

Recommendation: Versioning

One suggestion that might make things easier moving forward is to add some element of versioning.

As it is, when you pull down some ciphertext and decrypt your key, you just assume it was encrypted with the current algorithm. That's fine for now, but in time, the algorithm may change, and it would be difficult to tell what a given ciphertext was encrypted with.

The easiest way is to probably just to do something like prepend a byte to the ciphertext, like a 0, to indicate that we are using AES-256-GCM-with-random-padding. In the future, you can just increment that byte for more options, and add any additional structure you might need then.

Github Releases?

It would be nice to get release binaries via github, would you be interested in a pull request to add some release logic to the makefile?

In a similar way to how I have done it here https://github.com/wolfeidau/coffer/blob/master/Makefile.

Any other suggestions on how I could help with distribution of binaries are of course welcome.

Cheers

sneaker ls with panic: runtime error: invalid memory address or nil pointer dereference

I have sneaker 1.6 installed in a docker container:

sneaker version
version: 3e52e64
goversion: go version go1.6 linux/amd64
buildtime: 2016-05-26T17:50:18Z

with sneaker ls I get:
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x4d6682]

goroutine 1 [running]:
panic(0x8bbba0, 0xc82000e090)
/usr/local/go/src/runtime/panic.go:464 +0x3e6
github.com/aws/aws-sdk-go/service/s3.New(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/robotron/aws/go/src/github.com/codahale/sneaker/Godeps/_workspace/src/github.com/aws/aws-sdk-go/service/s3/service.go:41 +0x72
main.loadManager(0x7ea160)
/robotron/aws/go/src/github.com/codahale/sneaker/cmd/sneaker/main.go:222 +0x329
main.main()
/robotron/aws/go/src/github.com/codahale/sneaker/cmd/sneaker/main.go:55 +0x455

---

Tried to figure out the problem. Found the following hint in the aws-sdk-go/s3/session Info:
aws/aws-sdk-go@1a69d06

"...Migrating to this change you'll need to add a session.Session to your
application. The session will be used in each place a service client is
created. Naively this can be s3.New(session.New()) instead of the
previous s3.New(nil)...."_

In order to fix it I had to make the following changes:
1.)
in cmd/main.go I added:
import: "github.com/aws/aws-sdk-go/aws/session"

in loadManager I changed return in s3.New and kms.New to: nil -> session.New()

        return &sneaker.Manager{
                Objects: s3.New(session.New()),
                Envelope: sneaker.Envelope{
                        KMS: kms.New(session.New()),
                },
                Bucket:            u.Host,
                Prefix:            u.Path,
                EncryptionContext: ctxt,
                KeyId:             os.Getenv("SNEAKER_MASTER_KEY"),

2.) Provide AWS_REGION environment variable in addition to AWS_DEFAULT_REGION. - I think/hope that can be avoided calling proper aws-sdk-go api calls

Please check this out.

Decrypting KMS encrypted data

Hi,

If a got a piece of data encrypted with a KMS key, how can I use sneaker to decrypt it ?
Shall I just upload the file to the S3 bucket manually and use

sneaker download s3_path decrypted_file

?

Many thanks

Ken

git submdule for aws-sdk-go broken

This is the output I see:

go get -d -u github.com/codahale/sneaker
# cd /root/gopath/src/github.com/codahale/sneaker; git submodule update --init --recursive
fatal: no submodule mapping found in .gitmodules for path 'vendor/github.com/aws/aws-sdk-go'
package github.com/codahale/sneaker: exit status 128

sneaker download to STDOUT prints information other than the secret

I would think that the following would only output the secret to STDOUT and not the downloading status message.

Expected Behavior

% sneaker download my-secret -
this-is-the-secret                                                                                                                  

Current Behavior
Instead it prints a downloading status and then prints the secret.

% sneaker download my-secret -
2016/04/28 15:29:58 downloading -
this-is-the-secret

The documentation says:

Likewise, using - as the output path will make sneaker write the data to STDOUT. 
This allows you to pipe the output directly to a tar process, for example.

`sneaker ls <pattern>` stopped working

Hello, I've been experimenting with sneaker - quite a neat project! Initially, I tried using the sneaker ls <pattern> functionality, and it seemed to work well - however something happened and it stopped working. I'm not sure what changed - I was playing around with S3 object versioning (what happens if we delete all of the versions for an object, etc) in between - not sure if that's what did it.

Before:

$ AWS_DEFAULT_PROFILE=wave-dev aws-profile ./sneaker ls kraken/*
key                          modified          size  etag
kraken/DATADOG_API_KEY       2016-09-12T18:11  -18   f37f488706e962cdc6109250ac51c0b8
kraken/ENVIRONMENT           2016-09-12T18:11  -19   582ec2ba4011a0ac22671471874da14e
kraken/GITHUB_TOKEN          2016-09-12T17:46  -15   9057a9843327f0e749c7df1dfbf5e8f8
kraken/NEWRELIC_LICENSE_KEY  2016-09-12T18:10  -15   5f586cc2874ed3cc4b5ba91400fb0bef
kraken/THREATSTACK_KEY       2016-09-12T18:10  -13   61f72f312630b6bc50c745681903f54c

After:

$ AWS_DEFAULT_PROFILE=wave-dev aws-profile ./sneaker ls kraken/*
Usage:
  sneaker ls [<pattern>]
  sneaker upload <file> <path>
  sneaker download <path> <file>
  sneaker rm <path>
  sneaker pack <pattern> <file> [--key=<id>] [--context=<k1=v2,k2=v2>]
  sneaker unpack <file> <path> [--context=<k1=v2,k2=v2>]
  sneaker rotate [<pattern>]
  sneaker version

Issue compiling due to missing submodules

When attempting to compile this following the instructions you get an error. The error output is below. You are able to continue with the remaining steps but the go get does exit 1 so it makes it difficult to have performed in automation.

-> docker run -it golang bash
root@22a6cc1c9b32:/go# go get -d -u github.com/codahale/sneaker
# cd /go/src/github.com/codahale/sneaker; git submodule update --init --recursive
No submodule mapping found in .gitmodules for path 'vendor/github.com/aws/aws-sdk-go'
package github.com/codahale/sneaker: exit status 1

Unable to upload to S3 bucket which requires encryption

Looking in upload.go, it doesn't look like there is a way to specify that server side encryption is required for uploading to an S3 bucket.

https://github.com/jietang/sneaker/blob/3fcd6e491b55199d799712c6aafe1d4de007d539/upload.go#L26-L32

&s3.PutObjectInput{
    ContentLength: aws.Int64(int64(len(ciphertext))),
    ContentType:   aws.String(contentType),
    Bucket:        aws.String(m.Bucket),
    Key:           aws.String(fpath.Join(m.Prefix, path)),
    Body:          bytes.NewReader(ciphertext),
},

The param required would look something like this.

http://docs.aws.amazon.com/sdk-for-go/api/service/s3/#example_S3_PutObject

ServerSideEncryption:    aws.String("AES256")

I'll eventually open up a pull request and reference back to this issue. I just want this here to provide some context.

Works with --profile instead of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

It would be helpful to allow profile instead of strict access_key and secret.

make install failing

I don't see any mistakes I've made in following the instructions. I manually cloned the aws sdk repo (due to #24). When I run the make install it tells me the package is not a go package or is not in GOPATH, though as far as I can tell both are not true. See below:

root@a3dce834372b:~/gopath/sneaker# export GOPATH=/root/gopath
root@a3dce834372b:~/gopath/sneaker# make install
touch cmd/sneaker/version.go
/root/gopath/bin/govendor install  -ldflags "-X main.version '8573936' -X main.goVersion 'go version go1.6.2 linux/amd64' -X main.buildTime '2016-08-31T02:40:51Z'" +local
Error: Package "/root/gopath/sneaker" not a go package or not in GOPATH.
Makefile:16: recipe for target 'install' failed
make: *** [install] Error 2
root@a3dce834372b:~/gopath/sneaker# ll
total 180
drwxr-xr-x 5 root root  4096 Aug 30 19:25 ./
drwxr-xr-x 6 root root  4096 Aug 30 19:29 ../
drwxr-xr-x 8 root root  4096 Aug 31 02:40 .git/
-rw-r--r-- 1 root root     7 Aug 30 19:25 .gitignore
-rw-r--r-- 1 root root    81 Aug 30 19:25 .travis.yml
-rw-r--r-- 1 root root   595 Aug 30 19:25 LICENSE
-rw-r--r-- 1 root root   875 Aug 30 19:25 Makefile
-rw-r--r-- 1 root root 10535 Aug 30 19:25 README.md
-rw-r--r-- 1 root root 58585 Aug 30 19:25 architecture.png
drwxr-xr-x 3 root root  4096 Aug 30 19:25 cmd/
-rw-r--r-- 1 root root   794 Aug 30 19:25 download.go
-rw-r--r-- 1 root root  1520 Aug 30 19:25 download_test.go
-rw-r--r-- 1 root root  2997 Aug 30 19:25 envelope.go
-rw-r--r-- 1 root root   924 Aug 30 19:25 envelope_test.go
-rw-r--r-- 1 root root   719 Aug 30 19:25 fake_kms_test.go
-rw-r--r-- 1 root root  1215 Aug 30 19:25 fake_s3_test.go
-rw-r--r-- 1 root root  1263 Aug 30 19:25 list.go
-rw-r--r-- 1 root root  3659 Aug 30 19:25 list_test.go
-rw-r--r-- 1 root root  1032 Aug 30 19:25 pack.go
-rw-r--r-- 1 root root  2294 Aug 30 19:25 packaging_test.go
-rw-r--r-- 1 root root   348 Aug 30 19:25 rm.go
-rw-r--r-- 1 root root   611 Aug 30 19:25 rm_test.go
-rw-r--r-- 1 root root   664 Aug 30 19:25 rotate.go
-rw-r--r-- 1 root root  3202 Aug 30 19:25 rotate_test.go
-rw-r--r-- 1 root root  1554 Aug 30 19:25 sneaker.go
-rw-r--r-- 1 root root   485 Aug 30 19:25 unpack.go
-rw-r--r-- 1 root root   863 Aug 30 19:25 upload.go
-rw-r--r-- 1 root root  1998 Aug 30 19:25 upload_test.go
drwxr-xr-x 3 root root  4096 Aug 30 19:25 vendor/