sop is an editor of encrypted files that supports YAML, JSON and TEXT formats and encrypts with AWS KMS and PGP (via GnuPG). Watch the demo.
Table of Contents
First install some libraries from your package manager:
RHEL family:
sudo yum install libyaml-devel python-devel libffi-devel pip
Debian family:
sudo apt-get install libyaml-dev python-dev libffi-dev python-pip
MacOS:
brew install libffi libyaml sudo easy_install pip
Then install sops from pip:
sudo pip install --upgrade sops
Clone the repository, load the test PGP key and open the test files:
$ git clone https://github.com/mozilla/sops.git $ cd sops $ gpg --import tests/sops_functional_tests_key.asc $ sops example.yaml
This last step will decrypt example.yaml using the test private key. To create your own secrets files using keys under your control, keep reading.
Assuming you already have libffi and libyaml installed, the following commands will install sops in a virtualenv:
$ sudo pip install virtualenv --upgrade
$ virtualenv ~/sopsvenv
$ source ~/sopsvenv/bin/activate
$ pip install -U sops
$ sops -h | grep ^Version
Version 0.9
If you're using AWS KMS, create one or multiple master keys in the IAM console and export them, comma separated, in the SOPS_KMS_ARN env variable. It is recommended to use at least two master keys in different regions.
export SOPS_KMS_ARN="arn:aws:kms:us-east-1:656532927350:key/920aff2e-c5f1-4040-943a-047fa387b27e,arn:aws:kms:ap-southeast-1:656532927350:key/9006a8aa-0fa6-4c14-930e-a2dfb916de1d"
Your AWS credentials must be present in ~/.aws/credentials. sops uses boto3.
$ cat ~/.aws/credentials [default] aws_access_key_id = AKI..... aws_secret_access_key = mw......
If you want to use PGP, export the fingerprints of the public keys, comma separated, in the SOPS_PGP_FP env variable.
export SOPS_PGP_FP="85D77543B3D624B63CEA9E6DBC17301B491B3F21,E60892BB9BD89A69F759A1A0A3D652173B763E8F"
Note: you can use both PGP and KMS simultaneously.
Then simply call sops with a file path as argument. It will handle the encryption/decryption transparently and open the cleartext file in an editor.
$ sops mynewtestfile.yaml
mynewtestfile.yaml doesn't exist, creating it.
please wait while an encryption key is being generated and stored in a secure fashion
[... editing happens in vim, or whatever $EDITOR is set to ...]
file written to mynewtestfile.yaml
The resulting encrypted file looks like this:
myapp1: ENC[AES256_GCM,data:Tr7o=,iv:1=,aad:No=,tag:k=]
app2:
db:
user: ENC[AES256_GCM,data:CwE4O1s=,iv:2k=,aad:o=,tag:w==]
password: ENC[AES256_GCM,data:p673w==,iv:YY=,aad:UQ=,tag:A=]
# private key for secret operations in app2
key: |-
ENC[AES256_GCM,data:Ea3kL5O5U8=,iv:DM=,aad:FKA=,tag:EA==]
an_array:
- ENC[AES256_GCM,data:v8jQ=,iv:HBE=,aad:21c=,tag:gA==]
- ENC[AES256_GCM,data:X10=,iv:o8=,aad:CQ=,tag:Hw==]
- ENC[AES256_GCM,data:KN=,iv:160=,aad:fI4=,tag:tNw==]
sops:
kms:
- created_at: 1441570389.775376
enc: CiC....Pm1Hm
arn: arn:aws:kms:us-east-1:656532927350:key/920aff2e-c5f1-4040-943a-047fa387b27e
- created_at: 1441570391.925734
enc: Ci...awNx
arn: arn:aws:kms:ap-southeast-1:656532927350:key/9006a8aa-0fa6-4c14-930e-a2dfb916de1d
pgp:
- fp: 85D77543B3D624B63CEA9E6DBC17301B491B3F21
created_at: 1441570391.930042
enc: |
-----BEGIN PGP MESSAGE-----
hQIMA0t4uZHfl9qgAQ//UvGAwGePyHuf2/zayWcloGaDs0MzI+zw6CmXvMRNPUsA
...=oJgS
-----END PGP MESSAGE-----
A copy of the encryption/decryption key is stored securely in each KMS and PGP block. As long as one of the KMS or PGP method is still usable, you will be able to access you data.
To decrypt a file in a cat fashion, use the -d flag:
$ sops -d mynewtestfile.yaml
sops encrypted files contain the necessary information to decrypt their content. All a user of sops need is valid AWS credentials and the necessary permissions on KMS keys.
Given that, the only command a sops user need is:
$ sops <file>
<file> will be opened, decrypted, passed to a text editor (vim by default), encrypted if modified, and saved back to its original location. All of these steps, apart from the actual editing, are transparent to the user.
When creating a new files, sops uses the PGP and KMS defined in the command line arguments --kms and --pgp, or from the environment variables SOPS_KMS_ARN and SOPS_PGP_FP. That information is stored in the file under the sops section. When editing a file, it is trivial to add or remove keys: invoke sops with the flag -s to display the master keys while editing, and add or remove kms or pgp keys under the sops section.
For example, to add a KMS master key to a file, we would add the following entry:
sops:
kms:
- arn: arn:aws:kms:us-east-1:656532927350:key/920aff2e-c5f1-4040-943a-047fa387b27e
And, similarly, to add a PGP master key, we add its fingerprint:
sops:
pgp:
- fp: 85D77543B3D624B63CEA9E6DBC17301B491B3F21
When the file is saved, sops will update its metadata and encrypt the data key with the freshly added master keys. The removed entries are simply deleted from the file.
SOPS has the ability to use KMS in multiple AWS accounts by assuming roles in each account. Being able to assume roles is a nice feature of AWS that allows administrator to establish trust relationships between accounts, typically from the most secure account to the least secure one. In our use-case, we use roles to indicate that a user of the Master AWS account is allowed to make use of KMS master keys in development and staging AWS accounts. Using roles, a single file can be encrypted with KMS keys in multiple accounts, thus increasing reliability and ease of use.
You can use keys in various accounts by tying each KMS master key to a role that the user is allowed to assume in each account. The IAM roles documentation has full details on how this needs to be configured on AWS's side.
From the point of view of sops, you only need to specify the role a KMS key must assume alongside its ARN, as follows:
sops:
kms:
- arn: arn:aws:kms:us-east-1:656532927350:key/920aff2e-c5f1-4040-943a-047fa387b27e
role: arn:aws:iam::927034868273:role/sops-dev-xyz
The role must have permission to call Encrypt and Decrypt using KMS. An example policy is shown below.
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*",
"Principal": {
"AWS": [
"arn:aws:iam::927034868273:role/sops-dev-xyz"
]
}
}
You can specify a role in the --kms flag and SOPS_KMS_ARN variable by appending it to the ARN of the master key, separated by a + sign:
<KMS ARN>+<ROLE ARN> arn:aws:kms:us-west-2:927034868273:key/fe86dd69-4132-404c-ab86-4269956b4500+arn:aws:iam::927034868273:role/sops-dev-xyz
It is recommend to renew the data key on a regular basis. sops supports key rotation via the -r flag. A simple approach is to decrypt and reencrypt all files in place with rotation enabled:
for file in $(find . -type f -name "*.yaml"); do
sops -d -i $file
sops -e -i -r $file
done
The command below create a new file with a data key encrypted by KMS and PGP.
$ sops --kms "arn:aws:kms:us-west-2:927034868273:key/fe86dd69-4132-404c-ab86-4269956b4500" --pgp C9CAB0AF1165060DB58D6D6B2653B624D620786D /path/to/new/file.yaml
Similar to the previous command, we tell sops to use one KMS and one PGP key. The path points to an existing cleartext file, so we give sops flag -e to encrypt the file, and redirect the output to a destination file.
$ export SOPS_KMS+ARN="arn:aws:kms:us-west-2:927034868273:key/fe86dd69-4132-404c-ab86-4269956b4500"
$ export SOPS_PGP_FP="C9CAB0AF1165060DB58D6D6B2653B624D620786D"
$ sops -e /path/to/existing/file.yaml > /path/to/new/encrypted/file.yaml
Decrypt the file with -d.
$ sops -d /path/to/new/encrypted/file.yaml
Rather than redirecting the output of -e or -d, sops can replace the original file after encrypting or decrypting it.
# file.yaml is in cleartext
$ sops -e -i /path/to/existing/file.yaml
# file.yaml is now encrypted
$ sops -d -i /path/to/existing/file.yaml
# file.yaml is back in cleartext
sops primary use case is encrypting YAML and JSON configuration files, but it also has the ability to manage binary files. When encrypting a binary, sops will read the data as bytes, encrypt it, store the encrypted base64 under tree['data'] and write the result as JSON.
Note that the base64 encoding of encrypted data can actually make the encrypted file larger than the cleartext one.
In-place encryption/decryption also works on binary files.
$ dd if=/dev/urandom of=/tmp/somerandom bs=1024 count=512 512+0 records in 512+0 records out 524288 bytes (524 kB) copied, 0.0466158 s, 11.2 MB/s $ sha512sum /tmp/somerandom 9589bb20280e9d381f7a192000498c994e921b3cdb11d2ef5a986578dc2239a340b25ef30691bac72bdb14028270828dad7e8bd31e274af9828c40d216e60cbe /tmp/somerandom $ sops -e -i /tmp/somerandom please wait while a data encryption key is being generated and stored securely $ sops -d -i /tmp/somerandom $ sha512sum /tmp/somerandom 9589bb20280e9d381f7a192000498c994e921b3cdb11d2ef5a986578dc2239a340b25ef30691bac72bdb14028270828dad7e8bd31e274af9828c40d216e60cbe /tmp/somerandom
sops can extract a specific part of a YAML or JSON document, by provided the path in the --extract command line flag. This is useful to extract specific values, like keys, without needed an extra parser.
$ sops -d ~/git/svc/sops/example.yaml -t '["app2"]["key"]'
-----BEGIN RSA PRIVATE KEY-----
MIIBPAIBAAJBAPTMNIyHuZtpLYc7VsHQtwOkWYobkUblmHWRmbXzlAX6K8tMf3Wf
ImcbNkqAKnELzFAPSBeEMhrBN0PyOC9lYlMCAwEAAQJBALXD4sjuBn1E7Y9aGiMz
bJEBuZJ4wbhYxomVoQKfaCu+kH80uLFZKoSz85/ySauWE8LgZcMLIBoiXNhDKfQL
vHECIQD6tCG9NMFWor69kgbX8vK5Y+QL+kRq+9HK6yZ9a+hsLQIhAPn4Ie6HGTjw
fHSTXWZpGSan7NwTkIu4U5q2SlLjcZh/AiEA78NYRRBwGwAYNUqzutGBqyXKUl4u
Erb0xAEyVV7e8J0CIQC8VBY8f8yg+Y7Kxbw4zDYGyb3KkXL10YorpeuZR4LuQQIg
bKGPkMM4w5blyE1tqGN0T7sJwEx+EUOgacRNqM2ljVA=
-----END RSA PRIVATE KEY-----
The tree path syntax uses a regular python dictionary syntax, without the variable name. Extract keys by naming them, and array elements by numbering them.
$ sops -d ~/git/svc/sops/example.yaml -t '["an_array"][1]'
secretuser2
You can import sops as a module and use it in your python program.
import sops
pathtype = sops.detect_filetype(path)
tree = sops.load_file_into_tree(path, pathtype)
sops_key, tree = sops.get_key(tree)
tree = sops.walk_and_decrypt(tree, sops_key)
sops.write_file(tree, path=path, filetype=pathtype)
You most likely want to store encrypted files in a version controlled repository. Sops can be used with git to decrypt files when showing diffs between versions. This is very handy for reviewing changes or visualizing history.
To configure sops to decrypt files during diff, create a .gitattributes file at the root of your repository that contains a filter and a command.
... code:
*.yaml diff=sopsdiffer
Here we only care about YAML files. sopsdiffer is an arbitrary name that we map to a sops command in the git configuration file of the repository.
$ git config diff.sopsdiffer.textconv "sops -d"
$ grep -A 1 sopsdiffer .git/config
[diff "sopsdiffer"]
textconv = "sops -d"
With this in place, calls to git diff will decrypt both previous and current version of the target file prior to displaying the diff. And it even works with git client interfaces, because they call git diff under the hood!
sops only supports a subset of YAML's many types. Encrypting YAML files that contain strings, numbers and booleans will work fine.
Files that contain anchors will not work, because the anchors redefine that structure of the file at load time. sops uses the path to a value as additional data in the AEAD encryption, and thus dynamic paths generated by anchors break the authentication step.
JSON and TEXT file types have no such feature and do not suffer this limitation.
When sops creates a file, it generates a random 256 bits data key and asks each KMS and PGP master key to encrypt the data key. The encrypted version of the data key is stored in the sops metadata under sops.kms and sops.pgp.
For KMS:
sops:
kms:
- enc: CiC6yCOtzsnFhkfdIslYZ0bAf//gYLYCmIu87B3sy/5yYxKnAQEBAQB4usgjrc7JxYZH3SLJWGdGwH//4GC2ApiLvOwd7Mv+cmMAAAB+MHwGCSqGSIb3DQEHBqBvMG0CAQAwaAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAyGdRODuYMHbA8Ozj8CARCAO7opMolPJUmBXd39Zlp0L2H9fzMKidHm1vvaF6nNFq0ClRY7FlIZmTm4JfnOebPseffiXFn9tG8cq7oi
enc_ts: 1439568549.245995
arn: arn:aws:kms:us-east-1:656532927350:key/920aff2e-c5f1-4040-943a-047fa387b27e
For PGP:
sops:
pgp:
- fp: 85D77543B3D624B63CEA9E6DBC17301B491B3F21
created_at: 1441570391.930042
enc: |
-----BEGIN PGP MESSAGE-----
Version: GnuPG v1
hQIMA0t4uZHfl9qgAQ//UvGAwGePyHuf2/zayWcloGaDs0MzI+zw6CmXvMRNPUsA
pAgRKczJmDu4+XzN+cxX5Iq9xEWIbny9B5rOjwTXT3qcUYZ4Gkzbq4MWkjuPp/Iv
qO4MJaYzoH5YxC4YORQ2LvzhA2YGsCzYnljmatGEUNg01yJ6r5mwFwDxl4Nc80Cn
RwnHuGExK8j1jYJZu/juK1qRbuBOAuruIPPWVdFB845PA7waacG1IdUW3ZtBkOy3
O0BIfG2ekRg0Nik6sTOhDUA+l2bewCcECI8FYCEjwHm9Sg5cxmP2V5m1mby+uKAm
kewaoOyjbmV1Mh3iI1b/AQMr+/6ZE9MT2KnsoWosYamFyjxV5r1ZZM7cWKnOT+tu
KOvGhTV1TeOfVpajNTNwtV/Oyh3mMLQ0F0HgCTqomQVqw5+sj7OWAASuD3CU/dyo
pcmY5Qe0TNL1JsMNEH8LJDqSh+E0hsUxdY1ouVsg3ysf6mdM8ciWb3WRGxih1Vmf
unfLy8Ly3V7ZIC8EHV8aLJqh32jIZV4i2zXIoO4ZBKrudKcECY1C2+zb/TziVAL8
qyPe47q8gi1rIyEv5uirLZjgpP+JkDUgoMnzlX334FZ9pWtQMYW4Y67urAI4xUq6
/q1zBAeHoeeeQK+YKDB7Ak/Y22YsiqQbNp2n4CKSKAE4erZLWVtDvSp+49SWmS/S
XgGi+13MaXIp0ecPKyNTBjF+NOw/I3muyKr8EbDHrd2XgIT06QXqjYLsCb1TZ0zm
xgXsOTY3b+ONQ2zjhcovanDp7/k77B+gFitLYKg4BLZsl7gJB12T8MQnpfSmRT4=
=oJgS
-----END PGP MESSAGE-----
sops then opens a text editor on the newly created file. The user adds data to the file and saves it when done.
Upon save, sops browses the entire file as of a key/value tree. Every time sops encounters a leaf value (a value that does not have children), it encrypts the value with AES256_GCM using the data key and a 256 bits random initialization vector.
Each file uses a single data key to encrypt all values of a document, but each value receives a unique initialization vector and has unique authentication data.
Additional data is used to guarantee the integrity of the encrypted data and of the tree structure: when encrypting the tree, key names are concatenated into a byte string that is used as AEAD additional data (aad) when encrypting values encrypted. We expect that keys do not carry sensitive information, and keeping them in cleartext allows for better diff and overall readability.
Any valid KMS or PGP master key can later decrypt the data key and access the data.
Multiple master keys allow for sharing encrypted files without sharing master keys, and provide disaster recovery solution. The recommended way to use sops is to have two KMS master keys in different region and one PGP public key with the private key stored offline. If, by any chance, both KMS master keys are lost, you can always recover the encrypted data using the PGP private key.
In addition to authenticating branches of the tree using keys as additional data, sops computes a MAC on all the values to ensure that no value has been added or removed fraudulently. The MAC is stored encrypted with AES_GCM and the data key under tree->`sops`->`mac`.
Automating the distribution of secrets and credentials to components of an infrastructure is a hard problem. We know how to encrypt secrets and share them between humans, but extending that trust to systems is difficult. Particularly when these systems follow devops principles and are created and destroyed without human intervention. The issue boils down to establishing the initial trust of a system that just joined the infrastructure, and providing it access to the secrets it needs to configure itself.
In many infrastructures, even highly dynamic ones, the initial trust is established by a human. An example is seen in Puppet by the way certificates are issued: when a new system attempts to join a Puppetmaster, an administrator must, by default, manually approve the issuance of the certificate the system needs. This is cumbersome, and many puppetmasters are configured to auto-sign new certificates to work around that issue. This is obviously not recommended and far from ideal.
AWS provides a more flexible approach to trusting new systems. It uses a powerful mechanism of roles and identities. In AWS, it is possible to verify that a new system has been granted a specific role at creation, and it is possible to map that role to specific resources. Instead of trusting new systems directly, the administrator trusts the AWS permission model and its automation infrastructure. As long as AWS keys are safe, and the AWS API is secure, we can assume that trust is maintained and systems are who they say they are.
Using the AWS trust model, we can create fine grained access controls to Amazon's Key Management Service (KMS). KMS is a service that encrypts and decrypts data with AES_GCM, using keys that are never visible to users of the service. Each KMS master key has a set of role-based access controls, and individual roles are permitted to encrypt or decrypt using the master key. KMS helps solve the problem of distributing keys, by shifting it into an access control problem that can be solved using AWS's trust model.
When Mozilla's Services Operations team started revisiting the issue of distributing secrets to EC2 instances, we set the goal to store these secrets encrypted until the very last moment, when they need to be decrypted on target systems. Not unlike many other organizations that operate sufficiently complex automation, we found this to be a hard problem with a number of prerequisites:
- Secrets must be stored in YAML files for easy integration into hiera
- Secrets must be stored in GIT, and when a new CloudFormation stack is built, the current HEAD is pinned to the stack. (This allows secrets to be changed in GIT without impacting the current stack that may autoscale).
- Entries must be encrypted separately. Encrypting entire files as blobs makes git conflict resolution almost impossible. Encrypting each entry separately is much easier to manage.
- Secrets must always be encrypted on disk (admin laptop, upstream git repo, jenkins and S3) and only be decrypted on the target systems
SOPS can be used to encrypt YAML, JSON and BINARY files. In BINARY mode, the content of the file is treated as a blob, the same way PGP would encrypt an entire file. In YAML and JSON modes, however, the content of the file is manipulated as a tree where keys are stored in cleartext, and values are encrypted. hiera-eyaml does something similar, and over the years we learned to appreciate its benefits, namely:
- diff are meaningful. If a single value of a file is modified, only that value will show up in the diff. The diff is still limited to only showing encrypted data, but that information is already more granular that indicating that an entire file has changed.
- conflicts are easier to resolve. If multiple users are working on the same encrypted files, as long as they don't modify the same values, changes are easy to merge. This is an improvement over the PGP encryption approach where unsolvable conflicts often happen when multiple users work on the same file.
OpenPGP gets a lot of bad press for being an outdated crypto protocol, and while true, what really made look for alternatives is the difficulty to manage and distribute keys to systems. With KMS, we manage permissions to an API, not keys, and that's a lot easier to do.
But PGP is not dead yet, and we still rely on it heavily as a backup solution: all our files are encrypted with KMS and with one PGP public key, with its private key stored securely for emergency decryption in the event that we lose all our KMS master keys.
SOPS can be used without KMS entirely, the same way you would use an encrypted PGP file: by referencing the pubkeys of each individual who has access to the file. It can easily be done by providing sops with a comma-separated list of public keys when creating a new file:
$ sops --pgp "E60892BB9BD89A69F759A1A0A3D652173B763E8F,84050F1D61AF7C230A12217687DF65059EF093D3,85D77543B3D624B63CEA9E6DBC17301B491B3F21" mynewfile.yaml
The security of the data stored using sops is as strong as the weakest cryptographic mechanism. Values are encrypted using AES256_GCM which is the strongest symetric encryption algorithm known today. Data keys are encrypted in either KMS, which also uses AES256_GCM, or PGP which uses either RSA or ECDSA keys.
Going from the most likely to the least likely, the threats are as follow:
An attacker with access to an AWS console can grant itself access to one of the KMS master key used to encrypt a sops data key. This threat should be mitigated by protecting AWS accesses with strong controls, such as multi-factor authentication, and also by performing regular audits of permissions granted to AWS users.
PGP keys are routinely mishandled, either because owners copy them from machine to machine, or because the key is left forgotten on an unused machine an attacker gains access to. When using PGP encryption, sops users should take special care of PGP private keys, and store them on smart cards or offline as often as possible.
sops doesn't apply any restriction on the size or type of PGP keys. A weak PGP keys, for example 512 bits RSA, could be factorized by an attacker to gain access to the private key and decrypt the data key. Users of sops should rely on strong keys, such as 2048+ bits RSA keys, or 256+ bits ECDSA keys.
A vulnerability in AES256_GCM could potentially leak the data key or the KMS master key used by a sops encrypted file. While no such vulnerability exists today, we recommend that users keep their encrypted files reasonably private.
Mozilla Public License Version 2.0
- Julien Vehent <[email protected]>
- Daniel Thornton <[email protected]>
- Alexis Metaireau <[email protected]>
- Rémy Hubscher <[email protected]>
sops is inspired by hiera-eyaml, credstash , sneaker, password store and too many years managing PGP encrypted files by hand...