aws / aws-encryption-sdk-c

AWS Encryption SDK for C

AWS Encryption SDK for C

The AWS Encryption SDK for C is a client-side encryption library designed to make it easy for everyone to encrypt and decrypt data using industry standards and best practices. It uses a data format compatible with the AWS Encryption SDKs in other languages. For more information on the AWS Encryption SDKs in all languages, see the Developer Guide.

Also, see the API documentation for the AWS Encryption SDK for C.

Security issue notifications

See Support Policy for for details on the current support status of all major versions of this library.

Building the AWS Encryption SDK for C with support for AWS KMS

We will demonstrate some simple build recipes for Linux, Mac, and Windows operating systems. These recipes assume a newly installed system with default installs of dependency packages.

The Windows instructions install everything in your current directory (where you run the build process). To change the installation directory, see the Tips and Tricks section at the end of this README.

The AWS Encryption SDK for C can be used with AWS KMS, but it is not required. If you want to build a minimal version of the ESDK without AWS KMS support, see "Minimal C build without AWS KMS support", below.

For best results when doing a build with AWS KMS integration, do not install aws-c-common directly. Build and install the AWS SDK for C++, which will build and install aws-c-common for you (see the C++ SDK dependencies here). If you install aws-c-common before building the AWS SDK for C++, this will fool the AWS SDK for C++ install logic, and you will be forced to install several other dependencies manually. Version 1.8.32 of the AWS SDK for C++ is supported by version v1.0.1 of the AWS Encryption SDK for C.

If you are working on an EC2 instance, regardless of operating system

For best results, create an EC2 instance with an instance profile that at a minimum has AWS KMS permissions for Encrypt, Decrypt, and GenerateDataKey for at least one KMS key in your account. You will not need any other AWS permissions to use the AWS Encryption SDK for C.

Dependencies

1. OpenSSL 1.0.2 or newer, or 1.1.0 or newer
2. CMake 3.9 or newer
3. C/C++ compiler
4. aws-c-common, typically bundled with the AWS SDK for C++
5. The AWS SDK for C++ version 1.9.35 or newer

The AWS SDK for C++ and the AWS Encryption SDK for C share dependencies on OpenSSL, aws-c-common, and CMake. The AWS SDK for C++ has additional dependencies and prerequisites. See AWS SDK for C++: Getting Started.

You need to compile the AWS Encryption SDK for C and its dependencies as either all shared or all static libraries.

To build shared libraries, specify the -DBUILD_SHARED_LIBS=ON flag to build aws-c-common, the AWS SDK for C++, and the AWS Encryption SDK.

To build static libraries, specify the -DBUILD_SHARED_LIBS=OFF flag to build aws-c-common, the AWS SDK for C++, and the AWS Encryption SDK.

Linux Build Recipe

First, build the AWS SDK for C++. That will install the shared dependencies.

If you only need AWS SDK for C++ to use the AWS Encryption SDK, you have the option to build only the AWS KMS SDK. Add the -DBUILD_ONLY="kms" flag and -DBUILD_SHARED_LIBS=ON|OFF to cmake in the instructions provided.

Follow the AWS SDK for C++ build instructions.

Now, build and install the AWS Encryption SDK for C:

git clone --recurse-submodules https://github.com/aws/aws-encryption-sdk-c.git
mkdir build-aws-encryption-sdk-c && cd build-aws-encryption-sdk-c
cmake ../aws-encryption-sdk-c -DBUILD_SHARED_LIBS=ON
make && sudo make install ; cd ..

MacOS Build Recipe

Homebrew is a convenient way to get build tools for MacOS systems.

With Homebrew installed, run the following:

brew install [email protected] cmake

The AWS SDK for C++ can be installed with Homebrew, which will install the full AWS SDK. If you need the AWS SDK for C++ only to use the AWS Encryption SDK, you can build only the AWS KMS SDK. See these directions and specify the -DBUILD_ONLY="kms" flag to cmake in the instructions provided.

Now, build and install the AWS Encryption SDK for C:

git clone --recurse-submodules https://github.com/aws/aws-encryption-sdk-c.git
mkdir build-aws-encryption-sdk-c && cd build-aws-encryption-sdk-c
cmake -G Xcode -DBUILD_SHARED_LIBS=ON -DOPENSSL_ROOT_DIR="/usr/local/opt/[email protected]" ../aws-encryption-sdk-c 
xcodebuild -target install; cd ..

Windows Build Recipe

Note: See the docker-images folder for some Windows build recipes that automate many of these steps.

Note: These instructions do not yet correctly build a statically-linked AWS Encryption SDK for C on Windows systems, i.e. with -DBUILD_SHARED_LIBS=OFF. We will update this README with instructions for that configuration soon.

Install Visual Studio version 15 or later with the Windows Universal C Runtime and Git for Windows.

Use the "x64 Native Tools Command Prompt" for all commands listed here. Run the following commands in the directory where you want to do the build and installation.

Install Microsoft vcpkg by following these directions. Note the path to your vcpkg.cmake tool. This path will be something like vcpkg\scripts\buildsystems\vcpkg.cmake and you will need to pass it as -DCMAKE_TOOLCHAIN_FILE in your builds.

Use vcpkg to install prerequisites:

vcpkg install curl:x64-windows openssl:x64-windows

You may also want to configure vcpkg for Visual Studio CMake projects.

Build the AWS SDK for C++. This installs the aws-c-common dependency, too. If you need the AWS SDK for C++ only to use the AWS Encryption SDK, you have the option to build only the AWS KMS SDK. Add the -DBUILD_ONLY=kms to build only the AWS KMS client.

Follow these instructions, using -DCMAKE_TOOLCHAIN_FILE as described in the vcpkg setup instructions. Add -DBUILD_SHARED_LIBS=ON for shared libraries or -DBUILD_SHARED_LIBS=OFF for static libraries.

Now, build and install the AWS Encryption SDK for C. Be sure to update -DCMAKE_TOOLCHAIN_FILE in the commands below.

Update -DCMAKE_PREFIX_PATH to the location of your AWS SDK for C++ and aws-c-common installations.

git clone --recurse-submodules https://github.com/aws/aws-encryption-sdk-c.git
mkdir build-aws-encryption-sdk-c && cd build-aws-encryption-sdk-c
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DCMAKE_TOOLCHAIN_FILE"=%cd%\..\vcpkg\scripts\buildsystems\vcpkg.cmake" -DCMAKE_PREFIX_PATH="C:/Program Files (x86)/aws-cpp-sdk-all;C:/Program Files (x86)/aws-cpp-sdk-all/lib/aws-c-common" ..\aws-encryption-sdk-c
cmake --build . && cmake --build . --target install && cd ..

Building C only, without AWS KMS support

If you don't need AWS KMS support, you can build the AWS Encryption SDK for C without the AWS SDK. This build is C-only with no C++ support required.

To build without AWS KMS support, follow the build instructions above for your platform, substituting aws-c-common for aws-sdk-cpp.

Doxygen Documentation

To build the documentation, you'll need doxygen installed. Check the downloads page or use your OS package manager.

Next, rerun the above cmake with a -DBUILD_DOC="ON" flag before building aws-encryption-sdk-c.

Finally, run either make doc_doxygen (Unix), MSBuild.exe .\doc_doxygen.vcxproj (Windows) or xcodebuild -scheme doc_doxygen (Mac) to generate the documentation.

Compiling your program using the AWS Encryption SDK for C

Once you have installed the AWS Encryption SDK for C, you are ready to start writing your own programs with it.

When doing a C compilation (not using the AWS KMS keyring) be sure to include the flags -lcrypto -laws-encryption-sdk -laws-c-common.

When doing a C++ compilation (using the AWS KMS keyring) be sure to include the flags -std=c++11 -lcrypto -laws-encryption-sdk -laws-encryption-sdk-cpp -laws-c-common -laws-cpp-sdk-kms -laws-cpp-sdk-core.

In the examples directory of this repo are several self-standing C and C++ files. On a successful build, these files will already be compiled in the examples subdirectory of the build directory. Additionally, if you want a quick test that you have built and installed the AWS Encryption SDK successfully, you can copy any of the example files out of that directory and build them yourself.

Here are sample command lines using gcc/g++ to build a couple of the example files, assuming that the libraries and headers have been installed where your system knows how to find them.

g++ -o string string.cpp -std=c++11 -lcrypto -laws-encryption-sdk -laws-encryption-sdk-cpp -laws-c-common -laws-cpp-sdk-kms -laws-cpp-sdk-core -laws-crt-cpp
gcc -o raw_aes_keyring raw_aes_keyring.c -lcrypto -laws-encryption-sdk -laws-c-common

Note that the C++ files using the AWS KMS keyring will require you to make sure that AWS credentials are set up on your machine to run properly.

Tips and tricks

When building aws-sdk-cpp, you can save time by only building the subcomponents needed by the AWS Encryption SDK for C with -DBUILD_ONLY="kms"

The -DENABLE_UNITY_BUILD=ON option will further speed up the aws-sdk-cpp build.

To enable debug symbols, set -DCMAKE_BUILD_TYPE=Debug at initial cmake time, or use ccmake . to update the configuration after the fact.

If desired, you can set the installations to happen in an arbitrary directory with -DCMAKE_INSTALL_PREFIX=[path to install to] as an argument to cmake.

If you set CMAKE_INSTALL_PREFIX for the dependencies, when building the AWS Encryption SDK for C you must either (1) set CMAKE_INSTALL_PREFIX to the same path, which will cause it to pick up the dependencies and install to the same directory or (2) set CMAKE_PREFIX_PATH to include the same path, which will cause it to pick up the dependencies but NOT install to the same directory.

You can also use -DOPENSSL_ROOT_DIR=[path to where openssl was installed] to make the build use a particular installation of OpenSSL.

By default the cmake for this project detects whether you have aws-sdk-cpp-core and aws-sdk-cpp-kms installed in a place it can find, and only if so will it build the C++ components, (enabling use of the AWS KMS keyring.) You can override the detection logic by setting -DBUILD_AWS_ENC_SDK_CPP=OFF to never build the C++ components or by setting -DBUILD_AWS_ENC_SDK_CPP=ON to require building the C++ components (and fail if the C++ dependencies are not found.)

CBMC

To learn more about CBMC and proofs specifically, review the training material here.

The verification/cbmc/proofs directory contains CBMC proofs.

In order to run these proofs you will need to install CBMC and other tools by following the instructions here.

License

This library is licensed under the Apache 2.0 License.

aws-encryption-sdk-c's Issues

Implement trailing signatures

code style

Add an end-to-end example and test for the materials cache

keyring trace

Example code

aws_thread_launch should accept a const struct aws_thread_options *

There's no reason the options field should be non-const.

Audit coverage gaps

Go through the coverage report and make sure we have coverage for everything other than "impossible" error paths.

Default implementation of MKP/MK

returning a random data key. Or possibly pushed into default CMM?

Make functions in test_crypto.h static inline (or move the definitions to a .c file)

Move requested_alg to the CMM

The default CMM should own algorithm selection

Improve session object creation and initialization

Comment from PR 37:

Currently, the user has to call aws_cryptosdk_session_new followed immediately by either aws_cryptosdk_session_init_decrypt or aws_cryptosdk_session_init_encrypt on the exact same session object. Because the call to aws_cryptosdk_session_init_decrypt or aws_cryptosdk_session_init_encrypt immediately re-zeroizes the entire structure, if the user were to make any attempts to configure the structure in between these two calls, that user's work would be lost. Therefore, there is no value to doing the "new" and the "init" actions in two separate API calls, and we only create the possibility for the user to make mistakes.

It also violates the standard in our code base that functions with "init" in the name initialize memory in an object that must later be freed with a call to a "clean_up" function.

Furthermore, the implementation is inefficient in that it ends up zeroizing the entire structure three times when it is first created before the user does anything with it. (Once at session.c:243 and then again in two separate calls to session_reset within the "new" and "init" functions.)

Please fix these issues by making aws_cryptosdk_session_new take in an argument specifying whether the session is for encrypt or decrypt and getting rid of the aws_cryptosdk_session_init_decrypt and aws_cryptosdk_session_init_encrypt functions.

Run CI in both Release and Debug configurations

Debug configuration will catch assert failures, while release mode will run heavier optimizations and may turn up issues around UB.

KMS MK/MKP

Add name prefix to session_decrypt/session_encrypt helpers

These functions should start with aws_cryptosdk_ to avoid name clashes in static libraries

improve aws_cryptosdk_err_init_strings()

1. make it idempotent in the same way aws_load_error_strings is
2. make it call aws_load_error_strings first
3. check how comparable functions are named in other AWS SDK libraries and see if it should be renamed for consistency

Clarify in README and header docs the status of the aws-encryption-sdk-cpp subcomponent

We should probably describe this as a temporary API, in place until we 1) have a pure-C KMS keyring and/or 2) have a full API (including session, etc) that matches C++ conventions.

make sure `make install` works

New format static MK

Implement caching CMM

Add windows bcrypt backend

Currently, windows builds depend on openssl for crypto primitives, these should use bcrypt instead

Rename signctx to something more descriptive

Make decryption_request.enc_context const

Limit exported symbols to public API functions

Implement a multi-MKP

As with the MultipleProviderFactory in Java, we should provide a sample MKP that combines multiple MKPs into one.

Mark variables `const` whenever possible

Doing so is useful for the purpose of defensive programming as well as potentially enabling additional compiler optimizations. My proposal is to gradually introduce it when new code is added/existing code is modified, but alternatively one could also envision a large-scale PR.

Reduce framework duplication in AES/RSA keyring tests

See comment at #81 (comment)

Fix potential integer overflow in aws_cryptosdk_hdr_size

Header parser accepts incorrect headers

From the documentation (https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/message-format.html), it seems that when the encryption context is empty, a correct header should not contain a nonzero AAD Length. The header parsing function aws_cryptosdk_hdr_parse_init is able to parse a header with an empty encryption context and nonzero AAD Length successfully.
Here is an example where the bytes parsed and written can be different because of this issue:

#include <aws/cryptosdk/private/header.h>
#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#define BUF_SIZE sizeof(bytes)

uint8_t bytes [] = { 0x01, 0x80, 0x00, 0x14, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
                     0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x00, 0x02,
                     0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x0c, 0x02,
                     0x02, 0x02, 0x02, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
                     0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
                     0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01 };

int main(void) {
    struct aws_cryptosdk_hdr *hdr = malloc(sizeof(*hdr));
    struct aws_allocator *allocator = aws_default_allocator();

    uint8_t *src = bytes;
    uint8_t *out = malloc(BUF_SIZE);
    if (out) {
        // start with out and src buffers the same
        memcpy(out, src, BUF_SIZE);
        size_t bytes_written;
        // parse bytes
        assert(aws_cryptosdk_hdr_parse_init(allocator, hdr, src, BUF_SIZE) == AWS_OP_SUCCESS);
        // write bytes
        assert(aws_cryptosdk_hdr_write(hdr, &bytes_written, out, BUF_SIZE) == AWS_OP_SUCCESS);
        int i;
        // print differences between parsed and written bytes
        for (i = 0; i < BUF_SIZE; i++) {
            if (src[i] != out[i]) {
                printf("Difference at byte %d\n", i); 
            }   
        }   
        assert(hdr->aad_count == 0); // encryption context is empty
        /* The original bytes in src have aad_len of 2 and an aad_count of 0.
           Meanwhile, out has an aad_len of 0 (because the encryption context is empty). */
    }   

    free(hdr);
    free(out);
    return 0;
}

cleanup of materials.h

Currently there are structs in materials.h that we are not using (private key/public key) and a TODO comment about requested alg. These need to be cleaned up before release.

Revisit whether to use non-ARN key IDs in KMS keyring

KMS discovery keyring

change zero data key to something nonzero

Hmm... we might want to use a known test pattern instead of all-zero, to avoid false successes if there's uninitialized memory here.

(will require generating new test vectors)

Revisit build system of CBMC batch

The files of the form .cbmc-batch/jobs/openssl_*/Makefile are highly redundant. Examine whether there is a good way to refactor this build system and consolidate this.

JCEMasterKey-compatible static MK

cleanup logging in KMS keyring

Logging in KMS keyring calls some things errors that are not errors.

It also buffers up error messages and emits them together, which is not an ideal design.

Revisit this.

name the "things on a keyring"

Originally posted by @mattsb42-aws in #132 (comment)

Refactor auth_tag confusion in header structures and cipher routines

Currently, some areas of the code use 'authtag' to refer to the combined IV + GCM tag structure, while others use it to refer to just the GCM tag. We should refactor to use a consistent vocabulary here (and possibly combine the IV+tag fields in the header structure).

CBMC Proofs for HKDF

This is in relation to the HKDF tests and CBMC proofs. Looks like the CBMC tests written for HKDF are specific to the APIs introduced in v 1.1.1 of openssl which we donot use.

Note that our library supports HKDF for openssl versions 1.0.2+. We currently use the HKDF APIs introduced in v 1.1.0 which is significantly different from v 1.1.1 in that HKDF modes were introduced later, and for versions less than 1.1.0 we have our own implementation of RFC 5869 which we use. We definitely would like to have the proofs for all HKDF versions.

Testing compatibility of AWS Encryption SDK with known-good static test files

The known-good static test files are present in https://github.com/awslabs/aws-encryption-sdk-test-vector.

Get windows builds working

Fix KMS keyring generate functionality

currently only generates one EDK from first CMK, doesn't reencrypt with other CMKs

Document coding/code formatting standards

The code base employs a number of coding rules only known among the core development team. Explicitly documenting these will help people reading the code as well as new/external contributors.

Cross-compatibility testing with python

determine memory model and handling of encryption context by CMM/MKP/MK

Fix shared library builds on windows

improve or remove tests/integration/test_crypto.h harness

Now that we have reference counting, we do not need a struct to carry around sessions and CMMs together. Either refactor what is in this file or get rid of it entirely.

Doxygen docs public URL to link to README

While working with @juneb on ducmentation for AWS Encryption SDK for C, we noticed the need for providing the doxgen docs on a public URL for our customers to access.

Add negative tests for RSA/AES-GCM decryption

These tests should attempt to decrypt a mangled ciphertext and check that AWS_CRYPTOSDK_ERR_BAD_CIPHERTEXT is appropriately raised.

Decrypt operation on MK

We need a decrypt operation because, if a user wants to decrypt with a specific master key, they will need to adapt the MK into a MKP, and the adapter won't have anything to delegate decrypt to on the MK.

Change refcount down to release ordering

relaxed ordering does not prevent memory reordering after the decrement to zero