Coder Social home page Coder Social logo

aws-dynamodb-encryption-python's Introduction

Amazon DynamoDB Encryption Client for Python

Latest Version Supported Python Versions Code style: black Documentation Status tests static analysis

The Amazon DynamoDB Encryption Client for Python provides client-side encryption of Amazon DynamoDB items to help you to protect your table data before you send it to DynamoDB. It provides an implementation of the Amazon DynamoDB Encryption Client that is fully compatible with the Amazon DynamoDB Encryption Client for Java.

You can find the latest Python documentation at Read the Docs and you can find the latest full documents in our primary documents.

You can find our source on GitHub.

Security issue notifications

See Support Policy for details on the current support status of all major versions of this library.

Getting Started

Required Prerequisites

  • Python 3.8+

Installation

Note

If you have not already installed cryptography, you might need to install additional prerequisites as detailed in the cryptography installation guide for your operating system.

$ pip install dynamodb-encryption-sdk

Concepts

For a detailed description of the concepts that are important to understand when using this client, please review our Concepts Guide.

Usage

Helper Clients

We provide helper clients that look and feel like the low level client (EncryptedClient), service resource (EncryptedResource), and table resource (EncryptedTable) available from the boto3 library. For most uses, once configured, these clients can be used exactly as you would a standard client from boto3, and your items will be transparently encrypted on write and decrypted on read.

What can't I do with the helper clients?

For most uses, the helper clients (once configured) can be used as drop-in replacements for the boto3 clients. However, there are a couple cases where this is not the case.

Update Item

Because we can't know that a partial update you might be making to an item covers all of the signed attributes in your item, we do not allow update_item on the helper clients.

This is because if you update only some of the signed attributes, then next time you try to read that item the signature validation will fail.

Attribute Filtering

Because we can't know what attributes in an item are signed, the helper clients do not allow any attribute filtering.

For get_item, batch_get_item, and scan, this includes the use of AttributesToGet and ProjectionExpression.

For scan, this also includes the use of Select values SPECIFIC_ATTRIBUTES and ALL_PROJECTED_ATTRIBUTES.

This is because if you do not retrieve all signed attributes, the signature validation will fail.

Item Encryptor

The helper clients provide a familiar interface but the actual item encryption and decryption is handled by a low-level item encryptor. You usually will not need to interact with these low-level functions, but for certain advanced use cases it can be useful.

If you do choose to use the item encryptor functions directly, you will need to provide a CryptoConfig for each call.

>>> from dynamodb_encryption_sdk.encrypted.item import decrypt_python_item, encrypt_python_item
>>> plaintext_item = {
...     'some': 'data',
...     'more': 5
... }
>>> encrypted_item = encrypt_python_item(
...     item=plaintext_item,
...     crypto_config=my_crypto_config
... )
>>> decrypted_item = decrypt_python_item(
...     item=encrypted_item,
...     crypto_config=my_crypto_config
... )

When should I use the item encryptor?

One example of a use case where you might want to use the item encryptor directly is when processing items in a DynamoDB Stream. Since you receive the items data directly, and in DynamoDB JSON format, you can use the decrypt_dynamodb_item function to decrypt the item in the stream. We also provide helper transformation functions

Advanced Use

By default, the helper clients use your attribute actions and cryptographic materials provider to build the CryptoConfig that is provided to the item encryptor. For some advanced use cases, you might want to provide a custom CryptoConfig for specific operations.

All data plane operations (get item, put item, etc) on helper clients accept a crypto_config parameter in addition to all of the parameters that the underlying boto3 client accepts.

If this parameter is supplied, that CryptoConfig will be used for that operation instead of the one that the client would normally construct for you.

>>> from dynamodb_encryption_sdk.encrypted.table import EncryptedTable
>>> encrypted_table = EncryptedTable(
...     table=table,
...     materials_provider=my_crypto_materials_provider
... )
>>> encrypted_table.put_item(
...     Item=my_standard_item
... )  # this uses the crypto config built by the helper
>>> encrypted_table.put_item(
...     Item=my_special_item,
...     crypto_config=my_special_crypto_config
... )  # this uses my_special_crypto_config

aws-dynamodb-encryption-python's People

Contributors

ajewellamz avatar ajw-aws avatar alex-chew avatar bdonlan avatar david-koenig avatar dependabot[bot] avatar farleyb-amazon avatar gary-xero avatar imabhichow avatar johnwalker avatar josecorella avatar khubaibalam2000 avatar lavaleri avatar lizroth avatar mattsb42-aws avatar reaperhulk avatar ritvikkapila avatar robin-aws avatar salusasecondus avatar seebees avatar shubhamchaturvedi7 avatar texastony avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aws-dynamodb-encryption-python's Issues

validate underlying clients in helpers by shape, not class

@praus brought up the very good point that we might not always actually be getting a botocore client in the high level helpers (EncryptedTable/EncryptedResource/EncryptedClient). A good example of this is the DAX client.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DAX.client.run-application-python.html

We should change the validator for these to look at the shape of the underlying clients rather than their class type.

Return plaintext values in ReturnValues field in response

I just found out that when I do a delete_item operation with ReturnValues='ALL_OLD', the entries in the Attributes dict of the response from dynamodb are not decrypted. Looking at the dynamodb encryption sdk code, it looks like delete_item is not handled.

It's a minor issue, but I was wondering if decryption support for 'ALL_OLD' attributes in delete_item responses is planned to be added.

add option to cache encrypted items if unprocessed in batch request

Following on from #91 and #92, we now return the unprocessed items as plaintext items.

This lets users handle unprocessed items in a sane way without requiring them or their logic to know that they are using an encrypted client. However, it does mean that every unprocessed item will then be re-encrypted when the next write attempt is made, resulting in multiple CMP encryption_materials calls for each item.

We should determine what the best approach is to abstract this from the user: let them still use the same interface as they always have, but provide a configuration option to have the client hold onto the encrypted versions of those unprocessed items and drop them into subsequent write requests without needing to re-encrypt them.

Clean up linting

There is a variety of minor linting issues that fell through the gaps prior to launch. This issue will track fixing these issues.

No tests for query operations

We don't have any tests for the query operations. :(

  • Functional tests
  • Integration tests with ephemeral providers
  • Integration tests with KMS provider
  • All of the above for both EncryptedPaginator query operations

AWS KMS CMP does not provide hook to enforce key ID on decrypt

Unable to use dynamodb-encryption-sdk as layer

When dynamodb-encryption-sdk is added as a layer the function cannot find the module.

Steps to reproduce

  1. create zip of dynamodb-encryption-sdk on an ec2 instance
  2. use zip to create a lambda layer
  3. use layer in a lambda function
    from dynamodb_encryption_sdk.encrypted.table import EncryptedTable
    or any other dynamodb-encryption-sdk related import

The function throws an error
Unable to import module 'app': No module named 'dynamodb_encryption_sdk'

Is it not possible to use this package as a layer?

Return plaintext values in UnprocessedItems in batch_write_item response

Related to #90 and #91, in order for our encrypted helpers to function as drop-in replacements, we need to decrypt the contents of UnprocessedItems when we receive them in response to a batch_write_item operation.

In order to avoid hitting the CMP twice for every item, we should retain the plaintext key materials for use in processing the response, in the form of an ephemeral CMP that will be discarded once that request is handled.

Invalid transformation format:AES/256/CBC/PKCS5Padding : Interoperabilty between Python and Java SDKs

I tried a small custom application which encrypts and writes records to DDB using the Python SDK. The code (copied from here) is as following:

    aws_kms_cmp = AwsKmsCryptographicMaterialsProvider(key_id=aws_cmk_id)

    meta_table_name = 'metaStoreTable'

    # A name for the Most Recent Provider; used to
    # identify its CMPs in provider store storage
    material_name = 'testMRP'

    # Create an internal DynamoDB table resource for the MetaStore
    meta_table = boto3.resource('dynamodb').Table(meta_table_name)

    # Create the MetaStore with the Direct KMS Provider
    meta_store = MetaStore(
        table=meta_table,
        materials_provider=aws_kms_cmp
    )

    # Create a Most Recent Provider using the MetaStore
    most_recent_cmp = MostRecentProvider(
        provider_store=meta_store,
        material_name=material_name,
        version_ttl=60.0  # Check for a new version every 60 seconds
    )

    actions = AttributeActions(
        default_action=CryptoAction.SIGN_ONLY,
        attribute_actions={'cSnippet': CryptoAction.ENCRYPT_AND_SIGN}
    )

    encrypted_table = EncryptedTable(
        table=table,
        materials_provider=most_recent_cmp,
        attribute_actions=actions
    )
    encrypted_table.put_item(Item=result)

Java code to read records is below (taken from here)

            final AmazonDynamoDB dynamoDbClient = AmazonDynamoDBClientBuilder.standard().withRegion(REGION).build();

            final AWSKMS awskms = getOrCreateKMSClientBuilder().build();
            final String kmsKey = System.getenv(DYNAMODB_KMS_KEY); //same key arn as Python code

            final DirectKmsMaterialProvider directKmsMaterialProvider = new DirectKmsMaterialProvider(awskms, kmsKey);
            final DynamoDBEncryptor dynamoDBEncryptor = DynamoDBEncryptor.getInstance(directKmsMaterialProvider);

            final String keyTableName = "metaStoreTable"; //for MetaStore's internal table
            final long ttlInMillis = 60000;
            final String materialName = "testMRP";
            final MetaStore metaStore = new MetaStore(dynamoDbClient, keyTableName, dynamoDBEncryptor);
            final MostRecentProvider cmp = new MostRecentProvider(metaStore, materialName, ttlInMillis);

            final DynamoDBMapperConfig mapperConfig = DynamoDBMapperConfig.builder()
                    .withSaveBehavior(DynamoDBMapperConfig.SaveBehavior.PUT)
                    .withConsistentReads(DynamoDBMapperConfig.ConsistentReads.EVENTUAL)
                    .build();

            final DynamoDBMapper dynamoDBMapper = new DynamoDBMapper(dynamoDbClient,
                    mapperConfig, new AttributeEncryptor(cmp));

            dynamoDBMapper.queryPage(tableName, queryExpression);

Exception Traceback:

Caused by: java.lang.IllegalArgumentException: Algorithm does not exist
--
at com.amazonaws.services.dynamodbv2.datamodeling.encryption.DynamoDBEncryptor.lambda$static$0(DynamoDBEncryptor.java:69) ~[aws-dynamodb-encryption-java-1.14.1.jar:?]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_201]
at com.amazonaws.services.dynamodbv2.datamodeling.encryption.DynamoDBEncryptor.getBlockSize(DynamoDBEncryptor.java:396) ~[aws-dynamodb-encryption-java-1.14.1.jar:?]
at com.amazonaws.services.dynamodbv2.datamodeling.encryption.DynamoDBEncryptor.actualDecryption(DynamoDBEncryptor.java:380) ~[aws-dynamodb-encryption-java-1.14.1.jar:?]
at com.amazonaws.services.dynamodbv2.datamodeling.encryption.DynamoDBEncryptor.decryptRecord(DynamoDBEncryptor.java:283) ~[aws-dynamodb-encryption-java-1.14.1.jar:?]
at com.amazonaws.services.dynamodbv2.datamodeling.AttributeEncryptor.untransform(AttributeEncryptor.java:100) ~[aws-dynamodb-encryption-java-1.14.1.jar:?]
... 26 more
Caused by: java.security.NoSuchAlgorithmException: Invalid transformation format:AES/256/CBC/PKCS5Padding
at javax.crypto.Cipher.tokenizeTransformation(Cipher.java:321) ~[?:1.8.0_201]
at javax.crypto.Cipher.getTransforms(Cipher.java:428) ~[?:1.8.0_201]
at javax.crypto.Cipher.getInstance(Cipher.java:502) ~[?:1.8.0_201]
at com.amazonaws.services.dynamodbv2.datamodeling.encryption.DynamoDBEncryptor.lambda$static$0(DynamoDBEncryptor.java:66) ~[aws-dynamodb-encryption-java-1.14.1.jar:?]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_201]
at com.amazonaws.services.dynamodbv2.datamodeling.encryption.DynamoDBEncryptor.getBlockSize(DynamoDBEncryptor.java:396) ~[aws-dynamodb-encryption-java-1.14.1.jar:?]
at com.amazonaws.services.dynamodbv2.datamodeling.encryption.DynamoDBEncryptor.actualDecryption(DynamoDBEncryptor.java:380) ~[aws-dynamodb-encryption-java-1.14.1.jar:?]
at com.amazonaws.services.dynamodbv2.datamodeling.encryption.DynamoDBEncryptor.decryptRecord(DynamoDBEncryptor.java:283) ~[aws-dynamodb-encryption-java-1.14.1.jar:?]
at com.amazonaws.services.dynamodbv2.datamodeling.AttributeEncryptor.untransform(AttributeEncryptor.java:100) ~[aws-dynamodb-encryption-java-1.14.1.jar:?]

Python SDK version for the client: 1.2.0
Java SDK version for the client: 1.14.1

downstream tests in upstream library failing but upstream tests passing

As identified in pyca/cryptography#4979, some recent changes to the integration test suite setup caused pyca/cryptography's downstream tests to start failing[1].

This should have been caught in our upstream tests, but it was not.

There are two issues here:

  1. Our upstream tests are passing through environment variable configurations, which is masking the errors.
  2. The integration test parameterization setup is calling functions that require that environment variables are set during test collection, when those functions should only be called during test execution.

#1 is simple enough to fix, and that revealed #2[2].

The root of this is that while fixtures (such as integration_test_utils.cmk_arn) are executed when the test is executed, pytest_generate_tests is executed at test collection time.

[1] https://travis-ci.org/pyca/cryptography/jobs/579259777
[2]

============================================================================================ test session starts =============================================================================================
platform darwin -- Python 2.7.13, pytest-4.6.5, py-1.8.0, pluggy-0.12.0
cachedir: .tox/test-upstream-requirements-py27/.pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/bullocm/git/aws-dynamodb-encryption-python/.hypothesis/examples')
rootdir: /Users/bullocm/git/aws-dynamodb-encryption-python, inifile: setup.cfg
plugins: hypothesis-4.34.0, cov-2.7.1, forked-1.0.2, mock-1.10.4, xdist-1.29.0
collected 8679 items / 4 errors / 7417 deselected / 1258 selected                                                                                                                                            

=================================================================================================== ERRORS ===================================================================================================
_________________________________________________________________________ ERROR collecting test/integration/encrypted/test_client.py _________________________________________________________________________
test/integration/encrypted/test_client.py:29: in pytest_generate_tests
    set_parameterized_kms_cmps(metafunc)
test/integration/integration_test_utils.py:52: in set_parameterized_kms_cmps
    inner_cmp = AwsKmsCryptographicMaterialsProvider(key_id=cmk_arn_value())
test/integration/integration_test_utils.py:37: in cmk_arn_value
    AWS_KMS_KEY_ID
E   ValueError: Environment variable "AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID" must be set to a valid KMS CMK ARN for integration tests to run
________________________________________________________________________ ERROR collecting test/integration/encrypted/test_resource.py ________________________________________________________________________
test/integration/encrypted/test_resource.py:29: in pytest_generate_tests
    set_parameterized_kms_cmps(metafunc)
test/integration/integration_test_utils.py:52: in set_parameterized_kms_cmps
    inner_cmp = AwsKmsCryptographicMaterialsProvider(key_id=cmk_arn_value())
test/integration/integration_test_utils.py:37: in cmk_arn_value
    AWS_KMS_KEY_ID
E   ValueError: Environment variable "AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID" must be set to a valid KMS CMK ARN for integration tests to run
_________________________________________________________________________ ERROR collecting test/integration/encrypted/test_table.py __________________________________________________________________________
test/integration/encrypted/test_table.py:29: in pytest_generate_tests
    set_parameterized_kms_cmps(metafunc)
test/integration/integration_test_utils.py:52: in set_parameterized_kms_cmps
    inner_cmp = AwsKmsCryptographicMaterialsProvider(key_id=cmk_arn_value())
test/integration/integration_test_utils.py:37: in cmk_arn_value
    AWS_KMS_KEY_ID
E   ValueError: Environment variable "AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID" must be set to a valid KMS CMK ARN for integration tests to run
____________________________________________________________________ ERROR collecting test/integration/material_providers/test_aws_kms.py ____________________________________________________________________
test/integration/material_providers/test_aws_kms.py:36: in pytest_generate_tests
    set_parameterized_kms_cmps(metafunc, require_attributes=False)
test/integration/integration_test_utils.py:52: in set_parameterized_kms_cmps
    inner_cmp = AwsKmsCryptographicMaterialsProvider(key_id=cmk_arn_value())
test/integration/integration_test_utils.py:37: in cmk_arn_value
    AWS_KMS_KEY_ID
E   ValueError: Environment variable "AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID" must be set to a valid KMS CMK ARN for integration tests to run

BatchWriter attempts to re-encrypt unprocessed items.

Using the current 1.0.5 packaged version of the code the BatchWriter returned by Table.batch_writer() will raise exceptions if there are any unprocessed items returned from a batch_write.

The cause appears to be the client re-encrypting the already encrypted unprocessed items that get returned, and in the process validating the attribute names against the reserved attributes which fails.

Replicating this is easy enough, use a client to put items into a table at a rate that causes the writes to be throttled. Once at least one item fails and gets returned as an unprocessed item it'll trigger that error.

EncryptedPaginators are not decrypting items when using KMS crypto materials provider

A customer reported issues with an EncryptedPaginator failing to decrypt when using the AWS KMS crypto materials manager.

In digging into this, I realized that we also do not test for this at all...

I have duplicated this issue by adding paginator tests to the integration test suite.

The issue appears to be that the KMS encryption context is correctly including the hash and sort attribute names on ecrypt, but not on decrypt.

This appears to affect both the scan and query paginators.

...
DEBUG    botocore.endpoint:endpoint.py:101 Making request for OperationModel(name=GenerateDataKey) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'TrentService.GenerateDataKey', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.9.216 Python/3.7.0 Darwin/16.7.0 Botocore/1.12.216 DynamodbEncryptionSdkPython/1.1.0'}, 'body': b'{"KeyId": "arn:aws:kms:us-west-2:658956600833:key/b3537ef1-d8dc-4780-9f5a-55776cbb2f7f", "NumberOfBytes": 32, "EncryptionContext": {"*amzn-ddb-env-alg*": "AES/256", "*amzn-ddb-sig-alg*": "HmacSHA256/256", "partition_attribute": "test_value", "sort_attribute": "2231.0001", "*aws-kms-table*": "DDBEC-test-resources-TestTable-HS6VNXM82B6J"}}', 'url': 'https://kms.us-west-2.amazonaws.com/', 'context': {'client_region': 'us-west-2', 'client_config': <botocore.config.Config object at 0x1111ffb38>, 'has_streaming_input': False, 'auth_type': None}}

...

DEBUG    botocore.endpoint:endpoint.py:101 Making request for OperationModel(name=Decrypt) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'TrentService.Decrypt', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.9.216 Python/3.7.0 Darwin/16.7.0 Botocore/1.12.216 DynamodbEncryptionSdkPython/1.1.0'}, 'body': b'{"CiphertextBlob": "AQEBAHhA84wnXjEJdBbBBylRUFcZZK2j7xwh6UyLoL28nQ+0FAAAAH4wfAYJKoZIhvcNAQcGoG8wbQIBADBoBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDDWTPzuaYLFaMum2QQIBEIA7FsxQ6CVMg66d3RKRXgfSvDOpLnLWZc/6qaPhLYhs7nU9l8Z7ROgvqCNg3lGEYc4VYegUKv285/y0xFU=", "EncryptionContext": {"*amzn-ddb-env-alg*": "AES/256", "*amzn-ddb-sig-alg*": "HmacSHA256/256", "*aws-kms-table*": "DDBEC-test-resources-TestTable-HS6VNXM82B6J"}}', 'url': 'https://kms.us-west-2.amazonaws.com/', 'context': {'client_region': 'us-west-2', 'client_config': <botocore.config.Config object at 0x1111ffb38>, 'has_streaming_input': False, 'auth_type': None}}


Intermittent success when decrypting using KMSMasterKey, on object encrypted using Multiple CMKs

import aws_encryption_sdk
from aws_encryption_sdk.key_providers.kms import KMSMasterKey

CMK_ARNS = {
 "A": "arn1",
 "B": "arn2"
}
plaintext=b'abdc'
context = {'purpose': 'encryption context'}

master_key_provider_A = aws_encryption_sdk.KMSMasterKeyProvider()
master_key_provider_A.add_master_key(CMK_ARNS['A'])
master_key_provider_A.add_master_key(CMK_ARNS['B'])

master_key_provider_B = KMSMasterKey(key_id=CMK_ARNS['B'])

encrypted_message, header = aws_encryption_sdk.encrypt(
  encryption_context=context,
  key_provider=master_key_provider_A,
  source=plaintext
)

decrypted_message, decrypted_header = aws_encryption_sdk.decrypt(
        key_provider=master_key_provider_B,
  source=encrypted_message
)

Decrypting using master_key_provider_B fails intermittently with:

aws_encryption_sdk.exceptions.IncorrectMasterKeyError: Provided data key provider
MasterKeyInfo(provider_id='aws-kms', key_info=b'$ARN2') does not match Master Key
provider MasterKeyInfo(provider_id='aws-kms', key_info=b'$ARN1')

The expected behavior is that decryption always succeeds, since B's CMK is among those used in A's.

AwsKmsCryptographicMaterialsProvider - __attrs_post_init__ overrides custom regional clients

Hi Team,

I would like to report an issue regarding the regional client injection into the AwsKmsCryptographicMaterialsProvider.

Package version used: v1.1.1

Anything unusual about your environment or deployment: The environment is a local development environment, docker-compose is used with two services: a python:3.7.4-buster container for running the Python code, and a nsmithuk/local-kms container for a local KMS mock.

Bug reproduction steps:

  1. Set up a local KMS mock container. You can use it either as a docker-compose service or as a standalone container. REGION is to be set to us-west-2.

  2. Set up a key in the local mock container as per the usage notes using the following YAML:

Keys:
  - Metadata:
      KeyId: bc436485-5092-42b8-92a3-0aa8b93536dc
    BackingKeys:
      - 5cdaead27fe7da2de47945d73cd6d79e36494e73802f3cd3869f1d2cb0b5d7a9

Aliases:
  - AliasName: alias/testing
    TargetKeyId: bc436485-5092-42b8-92a3-0aa8b93536dc
  1. Create a small piece of code which does for example an item insert using EncryptedTable, with the following configuration:
aws_cmk_id = 'arn:aws:kms:us-west-2:111122223333:key/bc436485-5092-42b8-92a3-0aa8b93536dc'
client = boto3.session.Session().client('kms', region_name='us-west-2', endpoint_url='http://kms-service:8080')
aws_kms_cmp = AwsKmsCryptographicMaterialsProvider(key_id=aws_cmk_id, regional_clients={'us-west-2': client})

Please feel free to replace http://kms-service:8080 with the appropriate URL based on your local environment.

  1. After the above a call to the EncryptedTable helper should use the client which was created in step 3, but that's not the case, the AwsKmsCryptographicMaterialsProvider's _regional_clients attribute is empty, and it falls back to the base KMS client and tries to reach out to AWS.

I was able to mitigate the issue by deleting lines 218-220 from material_providers/aws_kms.py's __attrs_post_init__ function:

self._regional_clients = (
    {}
)  # type: Dict[Text, botocore.client.BaseClient]  # noqa pylint: disable=attribute-defined-outside-init

but I am unsure if this is the correct solution.

Any help regarding the above is much appreciated.

ValueError: Invalid endpoint: https://kms..amazonaws.com

Hi,

When calling AwsKmsCryptographicMaterialsProvider:

aws_kms_cmp = AwsKmsCryptographicMaterialsProvider(key_id=aws_cmk_id)

... I get the following error:

 Traceback (most recent call last):
  File "dynamodb_writer_enc.py", line 79, in <module>
    encrypted_table.put_item(Item=plaintext_item)
  File "/usr/local/lib/python3.7/site-packages/dynamodb_encryption_sdk/internal/utils.py", line 264, in encrypt_put_item
    crypto_config=crypto_config.with_item(_item_transformer(encrypt_method)(ddb_kwargs['Item']))
  File "/usr/local/lib/python3.7/site-packages/dynamodb_encryption_sdk/encrypted/item.py", line 145, in encrypt_python_item
    encrypted_ddb_item = encrypt_dynamodb_item(ddb_item, crypto_config)
  File "/usr/local/lib/python3.7/site-packages/dynamodb_encryption_sdk/encrypted/item.py", line 70, in encrypt_dynamodb_item
    encryption_materials = crypto_config.encryption_materials()
  File "/usr/local/lib/python3.7/site-packages/dynamodb_encryption_sdk/encrypted/__init__.py", line 93, in encryption_materials
    return self.materials_provider.encryption_materials(self.encryption_context)
  File "/usr/local/lib/python3.7/site-packages/dynamodb_encryption_sdk/material_providers/aws_kms.py", line 510, in encryption_materials
    initial_material, encrypted_initial_material = self._generate_initial_material(encryption_context)
  File "/usr/local/lib/python3.7/site-packages/dynamodb_encryption_sdk/material_providers/aws_kms.py", line 370, in _generate_initial_material
    response = self._client(key_id).generate_data_key(**kms_params)
  File "/usr/local/lib/python3.7/site-packages/dynamodb_encryption_sdk/material_providers/aws_kms.py", line 259, in _client
    return self._add_regional_client(region)
  File "/usr/local/lib/python3.7/site-packages/dynamodb_encryption_sdk/material_providers/aws_kms.py", line 239, in _add_regional_client
    ).client('kms', config=self._user_agent_adding_config)
  File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 889, in create_client
    client_config=config, api_version=api_version)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 76, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 291, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 80, in get_client_args
    timeout=(new_config.connect_timeout, new_config.read_timeout))
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 239, in create_endpoint
    raise ValueError("Invalid endpoint: %s" % endpoint_url)
ValueError: Invalid endpoint: https://kms..amazonaws.com 

Wrong formatting in the deprecated python warning

Security issue notifications

If you discover a potential security issue in the Amazon DynamoDB Encryption Client we ask that you notify AWS Security via our vulnerability reporting page. Please do not create a public GitHub issue.

Problem:

).format(py_version[0], py_version[1], minimum_version[0], minimum_version[1], params["date"])

The minimum_version and date seem to have been swapped in the deprecated python warning.

Solution:

Putting the params["date"] before the minimum_version[0] should fix the issue.

Out of scope:

Is there anything the solution will intentionally NOT address?

TableInfo index refresh fails when secondary indexes are present

We don't actually use the secondary indexes for anything, but TableInfo is opportunistically collecting them when refreshing index data. Unfortunately, TableIndex objects are not hashable, we are currently storing them in a set, and apparently this slipped through the cracks in testing...so that fails.

>>> table_info.refresh_indexed_attributes(client)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bullocm/git/aws-dynamodb-encryption-python/.tox/py36-local-fast/lib/python3.6/site-packages/dynamodb_encryption_sdk/structures.py", line 385, in refresh_indexed_attributes
    self._secondary_indexes.add(TableIndex.from_key_schema(index['KeySchema']))
TypeError: unhashable type: 'TableIndex'

Finish mypy integration

The mypy integration for this codebase is not quite complete. We will be working to close this gap asap. This issue will track that effort.

wrapped CMP should not require wrapping keys: this should be in wrapped materials

For cases where you are only signing, the un/wrapping keys can be omitted entirely. In this case it behaves like a static provider that only has a signing key.

Currently, we check for the presence of un/wrapping keys before we create the wrapped materials. This check should be in the wrapped materials, when we ask for an encryption or decryption key.

Add Attested Materials provider

Hi,

I have been working a lot with Nitro enclaves. Currently implementing attestation into our application where we want to call DynamoDB from within the enclave and perform decryption with attestation.

Unfortunately, boto3 doesn't support passing the Receiptient param to the KMS API call.

This can be resolved by creating a new materials provider attestation support. I am working on a solution for this would be interested to see if this could be added?

Compatibility with AWS' automatic KMS key rotation?

We are using the Direct KMS Materials Provider to encrypt items in our dynamodb table with an AWS CMK. We were wondering what the impact of turning on automatic yearly key rotation in AWS for this CMK would be? Would the client & KMS be able to transparently encrypt/decrypt items created both before the rotation?

Ex. Assuming Application uses the AwsKmsCryptographicMaterialsProvider class
6/1/2022 Application uses CMK id 1234 to insert encrypted Item A into table
6/2/2022 CMK happens to be rotated automatically by AWS
6/3/2022 Application attempts to use CMK id 1234 to read Item A. Is this library able to transparently determine the correct key material used to decrypt Item A even with the AwsKmsCryptographicMaterialsProvider being initiated with just the ID of AWS KMS CMK?

We'd love to test this on our own but can't seem to trigger an automatic key rotation manually and don't want to wait a year to see if our assumption was correct :)

CPython 3.4 support

In the next release, pyca/cryptography is dropping support for CPython 3.4[1]. This library is what initially set our lower bound for CPython version support, and if they are dropping support for 3.4 we should consider dropping that support as well because we rely so heavily on them.

Based on our PyPI download stats this year and the last month, I think that this is a low-risk course of action.

Date range: 2019-01-01 - 2019-12-31

category percent downloads
2.7 90.10% 137,955
3.6 8.47% 12,966
3.7 0.85% 1,304
null 0.57% 866
3.5 0.01% 14
3.4 0.01% 9
3.8 0.00% 1
Total 153,115

Date range: 2019-11-01 - 2019-11-30

category percent downloads
2.7 86.56% 22,766
3.6 11.22% 2,951
3.7 1.86% 489
null 0.35% 93
3.5 0.00% 1
Total 26,300

[1] pyca/cryptography#5087

Support transactional methods in EncryptionClient

Problem:

Are not currently supported by the EncryptionClient and passed through to the underlying client.

Solution:

Fully implement transact_get_items() and transact_write_items() in the EncryptionClient

Out of scope:

Is there anything the solution will intentionally NOT address? No

Workaround

I was able to implement the following workaround to encrypt one of the Put requests within my transaction:

aws_kms_cmp = AwsKmsCryptographicMaterialsProvider(key_id=KEY_ARN)
actions = AttributeActions(
    default_action=CryptoAction.DO_NOTHING,
    attribute_actions={"access_token": CryptoAction.ENCRYPT_AND_SIGN},
)
encrypted_client = EncryptedClient(
    client=dynamodb.meta.client,
    materials_provider=aws_kms_cmp,
    attribute_actions=actions,
    expect_standard_dictionaries=True,
    auto_refresh_table_indexes=False
)

item = {
    "pk": f"USER#{user_id}#ITEM#{item_id}",
    "sk": "v0",
    "access_token": access_token,
    "institution_id": institution_id,
    "institution_name": institution.get("name"),
    "link_session_id": metadata.get("link_session_id"),
    "created_at": now,
}

def mock_write_method(**kwargs):
    return kwargs.get("Item")

encrypt_item = partial(
    encrypt_put_item,
    encrypted_client._encrypt_item,
    encrypted_client._item_crypto_config,
    mock_write_method,
)
encrypted_item = encrypt_item(TableName=TABLE_NAME, Item=item)

items = [
    {
        "Put": {
            "TableName": TABLE_NAME,
            "Item": encrypted_item
        }
    }
]
dynamodb_client.transact_write_items(TransactItems=items)

typehints and imports

Just wanted to let y'all know that you can remove stuff like this now that your minimum supported Python version is 3.7.

try: # Python 3.5.0 and 3.5.1 have incompatible typing modules
from typing import Dict, Iterable, List, Optional, Set, Text # noqa pylint: disable=unused-import
except ImportError: # pragma: no cover
# We only actually need these imports when running the mypy checks
pass

I put those in because the version of the typing library for specifically 3.5.0 and 3.5.1 only has a subset of the type definitions present for newer versions. Since the typehints are all in comments (another thing you can change now that you don't support 2.7 anymore), the imports were only necessary when running mypy, thus the except: pass rather than stubbing.

AwsKmsCryptographicMaterialsProvider Design/Behavior Improvements

Problem:

The correct behavior of the DirectKMSMaterialProvider is to use the configured CMK on encrypt, and to let KMS determine the correct CMK to use on Decrypt (this is a common KMS pattern, as KMS ciphertext stores the CMK used as metadata).

However, this behavior for DirectKMSMaterialProvider is potentially confusing, as customers may expect that the CMK configured on the CMP is also "used" to decrypt, and may be surprised if decryption succeeds even though the configured CMK was not the CMK used to encrypt the data.

Solution:

Since the original DirectKMSMaterialProvider was designed, KMS has introduced a keyId param on Decrypt that ensures the call fails if a different key was used to encrypt the ciphertext.

We should consider either updating or replacing the DirectKMSMaterialProvider to allow enforcing a particular key on decrypt, similar to the Strict vs. Discovery modes expressed by the AWS Encryption SDK's Keyrings and Master Key Providers.

Our new design should:

  • maintain API parity between the DDBEC for Python and Java.
  • be as simple as possible to reason about.
  • minimize possible "modes" for behavior, and ensure that any "mode" needs to be explicitly chosen by customers on config.
  • ensure that any default configuration/behavior chooses the safest/most conservative option for customers.

Getting null values in `desc` and `sign` fields when I fetch the item using get_item function in boto3

Problem:

Recently I was trying to encrypt my data and store it in dynamodb table, I used Item encryptor directly, and according to the information given in this doc aws dynamo db encryption client at the end it says that, To view the encrypted item, call the get_item method on the original table object, instead of the encrypted_table object. It gets the item from the DynamoDB table without verifying and decrypting it.

But when I fetched the data ,l this is what I got:

The goal was to encrypt the password

"Item": { "attribute1": "got the data", "password": null, "*amzn-ddb-map-desc*": null, "attribute2": "got the data", "*amzn-ddb-map-sig*": null, "attribute3": "got the data", "attribute4": "got the data" },

In the above result, the password , *amzn-ddb-map-desc*, *amzn-ddb-map-sig* are null, my crypto config is also same, I thought I would fetch this and decrypt with the inbuilt functions in encryption client, can anyone please help?

Upgrading "Do Nothing" model to one with a single action fails to decrypt old items

Problem:

According to our documentation it should always be possible to add new attributes to our model without issue: https://docs.aws.amazon.com/dynamodb-encryption-client/latest/devguide/data-model.html#add-attribute

However, if you start with data encrypted using

    actions = AttributeActions(
        default_action=CryptoAction.DO_NOTHING
    )

And update to using

    actions = AttributeActions(
        default_action=CryptoAction.DO_NOTHING, attribute_actions={"someNewField": CryptoAction.ENCRYPT_AND_SIGN}
    )

You run into issues. This is because data under the first model doesn't have a material description or signature written with it. Once the model is updated to include an action other than DO_NOTHING, it always expects there to be a material description and signature, even if the record it's attempting to decrypt doesn't include someNewField yet.

Solution:

We should probably update the logic here to also pass through if the item under decrypt specifically doesn't have attributes where encryption or signing is needed, even if the attributeActions includes an encrypt or sign action for a non-present field.

if crypto_config.attribute_actions.take_no_actions:
# If we explicitly have been told not to do anything to this item, just copy it.
return item.copy()

def __attrs_post_init__(self):
# () -> None
"""Determine if any actions should ever be taken with this configuration and record that for reference."""
for attribute in ReservedAttributes:
if attribute.value in self.attribute_actions:
raise ValueError('No override behavior can be set for reserved attribute "{}"'.format(attribute.value))
# Enums are not hashable, but their names are unique
_unique_actions = {self.default_action.name}
_unique_actions.update({action.name for action in self.attribute_actions.values()})
no_actions = _unique_actions == {CryptoAction.DO_NOTHING.name}
self.take_no_actions = no_actions # attrs confuses pylint: disable=attribute-defined-outside-init

aws_kms_encrypted_table.py giving "Failed to generate materials using AWS KMS"

Hi,

I created a sample encrypted table on AWS DynamoDB based on the example. In the table I created a partition_attribute and a sort_attribute. I also encrypted the table with KMS and then copied the KMS Master Key ARN to pass in to aws_cmk_id which is the argument in encrypt_item function in the python code. I am getting an error "Failed to generate materials using AWS KMS" am I doing something wrong? Do I need to do anything else to get this to work?

I have added the code I am trying to run below, which is basically the example and I just added a main function to pass in arguments.

# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
"""Example showing use of AWS KMS CMP with EncryptedTable."""
import os
import sys
import argparse
import getopt
import boto3
from boto3.dynamodb.types import Binary
import botocore.session

from dynamodb_encryption_sdk.encrypted.table import EncryptedTable
from dynamodb_encryption_sdk.identifiers import CryptoAction
from dynamodb_encryption_sdk.material_providers.aws_kms import AwsKmsCryptographicMaterialsProvider
from dynamodb_encryption_sdk.structures import AttributeActions


def encrypt_item(table_name, aws_cmk_id):
    """Demonstrate use of EncryptedTable to transparently encrypt an item."""
    index_key = {"partition_attribute": "is this", "sort_attribute": 55}
    plaintext_item = {
        "example": "data",
        "some numbers": 99,
        "and some binary": Binary(b"\x00\x01\x02"),
        "leave me": "alone",  # We want to ignore this attribute
    }
    # Collect all of the attributes that will be encrypted (used later).
    encrypted_attributes = set(plaintext_item.keys())
    encrypted_attributes.remove("leave me")
    # Collect all of the attributes that will not be encrypted (used later).
    unencrypted_attributes = set(index_key.keys())
    unencrypted_attributes.add("leave me")
    # Add the index pairs to the item.
    plaintext_item.update(index_key)
	
    session = boto3.Session(aws_access_key_id='xxxxxxxxxxxxxxxxxxxx',
                        aws_secret_access_key='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
                        region_name='ca-central-1')

    dynamodb = session.resource('dynamodb')
	
    # Create a normal table resource.
    table = dynamodb.Table(table_name)  # generated code confuse pylint: disable=no-member
    # Create a crypto materials provider using the specified AWS KMS key.
    aws_kms_cmp = AwsKmsCryptographicMaterialsProvider(key_id=aws_cmk_id)
    # Create attribute actions that tells the encrypted table to encrypt all attributes except one.
    actions = AttributeActions(
        default_action=CryptoAction.ENCRYPT_AND_SIGN, attribute_actions={"leave me": CryptoAction.DO_NOTHING}
    )
    # Use these objects to create an encrypted table resource.
    encrypted_table = EncryptedTable(table=table, materials_provider=aws_kms_cmp, attribute_actions=actions)

    # Put the item to the table, using the encrypted table resource to transparently encrypt it.
    encrypted_table.put_item(Item=plaintext_item)

    # Get the encrypted item using the standard table resource.
    encrypted_item = table.get_item(Key=index_key)["Item"]

    # Get the item using the encrypted table resource, transparently decyrpting it.
    decrypted_item = encrypted_table.get_item(Key=index_key)["Item"]

    # Verify that all of the attributes are different in the encrypted item
    for name in encrypted_attributes:
        assert encrypted_item[name] != plaintext_item[name]
        assert decrypted_item[name] == plaintext_item[name]

    # Verify that all of the attributes that should not be encrypted were not.
    for name in unencrypted_attributes:
        assert decrypted_item[name] == encrypted_item[name] == plaintext_item[name]

    # Clean up the item
    encrypted_table.delete_item(Key=index_key)
	
def main(argv):
	encrypt_item( 'TestTable', 'arn:aws:kms:ca-central-1:xxxxxxxxxxxx:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')

if __name__ == "__main__":
   main(sys.argv[1:])  

Generate non-conflicting frozen upstream test requirements

pyca/cryptography runs downstream tests for this project and other to verify that their changes have not broken identified downstream dependencies.

Today, an update to botocore broke moto, which broke this repository's tests, which broke the pyca/cryptography downstream tests. pyca/cryptography#4415

We need to define and automate creation of a special frozen test requirements file for upstream tests to use when testing this repository in order to avoid this happening again.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.