awslabs / dynamodb-transactions

Transactions for Amazon DynamoDB

[IMPORTANT] Since November 2018, DynamoDB offers transactional APIs, simplifying the developer experience of making coordinated, all-or-nothing changes to multiple items both within and across tables. DynamoDB Transactions provide atomicity, consistency, isolation, and durability (ACID) in DynamoDB, enabling you to maintain data correctness in your applications more easily. We strongly recommend all developers to use DynamoDB’s built-in, servers-side transactions instead of this client-side library. To learn more about DynamoDB Transactions, see https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transactions.html.

Amazon DynamoDB Client-Side Transactions enables Java developers to easily perform atomic writes and isolated reads across multiple items and tables when building high scale applications on Amazon DynamoDB. You can get started in minutes using Maven.

• Transactions Details & Design
• DynamoDB Forum
• Transactions Library Issues

The Amazon DynamoDB Client-Side Transactions library is built on top of the low-level Amazon DynamoDB client in the AWS SDK for Java. For support in using and installing the AWS SDK for Java, see:

• API Docs
• SDK Developer Guide
• AWS SDK Forum
• SDK Homepage
• Java Development AWS Blog

Features

• Atomic writes: Write operations to multiple items are either all go through, or none go through.
• Isolated reads: Read operations to multiple items are not interfered with by other transactions.
• Sweepers: In-flight transaction state is stored in a separate table, and convenience methods are provided to "sweep" this table for "stuck" transactions.
• Easy to use: Mimics the Amazon DynamoDB API by using the request and response objects from the low-level APIs, including PutItem, UpdateItem, DeleteItem, and GetItem.
• Table helpers: Includes useful methods for creating tables such as and waiting for them to become ACTIVE.

Getting Started

1. Sign up for AWS - Before you begin, you need an AWS account. Please see the AWS Account and Credentials section of the developer guide for information about how to create an AWS account and retrieve your AWS credentials.
2. Minimum requirements - To run the SDK you will need Java 1.6+. For more information about the requirements and optimum settings for the SDK, please see the Java Development Environment section of the developer guide.
3. Install the Amazon DynamoDB Transactions Library - Using Maven is the recommended way to install the Amazon DynamoDB Transactions Library and its dependencies, including the AWS SDK for Java. To download the code from GitHub, simply clone the repository by typing: git clone https://github.com/awslabs/dynamodb-transactions, and run the Maven command described below in "Building From Source".
4. Run the examples - The included TransactionExamples automatically creates the necessary transactions tables, an example table for data and executes several operations with transactions. You can run the examples using Maven by:
5. Ensure you have already built the library using Maven (see "Building From Source" below)
6. Change into the examples directory of the project
7. Add your AWS Credentials to the file: src/main/resources/com/amazonaws/services/dynamodbv2/transactions/examples/AwsCredentials.properties
8. Compile the subproject by typing: mvn clean install
9. Run the examples by typing: mvn exec:java -Dexec.mainClass="com.amazonaws.services.dynamodbv2.transactions.examples.TransactionExamples"

Building From Source

Once you check out the code from GitHub, you can build it using Maven. To disable the GPG-signing in the build, use: mvn clean install -Dgpg.skip=true

dynamodb-transactions's Issues

Please Please Please Support Requests with Conditions

I got through most of my project before I found out that requests with conditions weren't supported. This is an amazing library and it would be great if you all could support that one day.

Thanks!

Deploy to Maven Central

https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide

It's not that hard.

Transaction with async requests

Any plans on supporting async methods of AmazonDynamoDBAsyncClient?

Plans to support DynamoDBMapper?

Just wondering if there any plans to support DynamoDBMapper.

Future support

Is this lib going to be supported in future? We are planning to use this very useful lib in our production code and we need to know if this lib will be maintained in the future.

Does this library provide encryption?

db mapper can encrypt object by giving a "provider" when creating it.
but it seems that this library is not using dbmapper.

Thanks,

Question: Known Limitations

Hello,

today I found your library that appears promising for my use case.

I am not sure I understood the concept sufficiently.
Currently we are running against the native TX item size limitation (=25) of DynamoDB.

Can you tell if there is a similar limitation using your library?
As I understood the guide, that pending items are persisted in a helper table(?)
Does it mean, there is no limitation for the max number of items per transaction?

Thanks and best regards
Thomas

Add transaction time out and check if transaction has been timed out before deciding to roll back pending transactions

By rolling back the transaction keeping the lock to resolve the contention, the transactions get in the way of each other and make the one keeping the lock cannot complete its transaction.

For example, process A creates a transaction and starts putting requests to it. After that, process B comes in, creates a transaction, puts requests and finds out that the transaction of process A is keeping the lock. It then rolls back the transaction of process A and acquires the lock. Process A now has a rolled back transaction and cannot complete what it is doing.

Normally, it should be "first come, first serve". The transaction of process A should have the right to keep the lock for a defined amount of time. In that time, if any transaction tries to get the lock, it should be the one having to be rolled back, not the transaction of process A. After the time, the transaction of process A will be considered as timed out transaction, and everything (sweeper and other transactions) can rolled it back.

Upgrade to 400 Kb item limit?

The docs say transactions are limited to 64 Kb - e.g. https://github.com/awslabs/dynamodb-transactions/blob/master/DESIGN.md

As Amazon now supports 400 Kb items, are 400 Kb transactions supported (I.e. Dynamodb-transactions Docs need updating) or does the library need updating to support 400 Kb items in transactions?

Are there any issues with single-items conditional writes?

Hello,

I know this may sound as a silly question, but does your library support conditional writes on single objects, provided that those writes are included in a transaction?
Since you are storing temporary images of the objects not yet committed in a temporary table, I was wondering how this will affect the case of conditional writes.
What if within the same transaction a conditional write fails, I therefore retrieve the current state of the object and I retry to write? Would you envision any issue with that?

Thanks

Isolation Levels - Clarification

I'm utilizing the DynamoDB Transaction Library and studying the "IsolationLevel" enum to get a better understanding of how to perform isolated reads. Looking for documentation on the 3 available isolation levels but can't find anything online:

• UNCOMMITTED
• COMMITTED
• READ_LOCK

For anyone familiar with these IsolationLevels an explanation or a link to appropriate documentation would be much appreciated! I have 3 Dynamo tables each of which will always be updated within a single transaction using the TransactionManager and I'm looking to read the state of a collection of items from all 3 tables post-transaction.

unlockItemAfterCommit should add ConditionExpression that "Applied" is true for Put and Update requests

Logically speaking, a prerequisite for unlocking items after a commit, is for all changes to be applied.
Therefore, the should be a ConditionExpression in

https://github.com/awslabs/dynamodb-transactions/blob/master/src/main/java/com/amazonaws/services/dynamodbv2/transactions/Transaction.java#L649

that checks that AttributeName.APPLIED is true.

Without this ConditionExpression there can be race conditions when there are resumed transactions that try to add requests instead of always rolling back, or put differently, there are multiple coordinators working on the same transaction.
In particular, there might be the case, which is similar to #27 , where:

• transaction1 contains 5 requests that have been applied.
• coordinator1 reads transaction1 and thinks it contains 5 requests.
• coordinator2 adds a 6th request to transaction1 and locked the reference item, but has yet to apply it.
• coordinator1 marks transaction1 as committed.
• coordinator2 fails to apply the 6th request.
• coordinator1 releases the lock on the item referenced by the 6th request.
• Error: transaction1 says the all 6 requests have been committed, whereas in reality only the first 5 have.

This seems to suggest that not only do we need the the check on APPLIED, but we also need to ensure that there can be only a single coordinator that adds requests to transactions. In other words, all resumed transactions or all coordinators except the initiator should be only allowed to rollback transactions.

Allow put/update to overwrite previous delete

Hi!

It is currently forbidden to perform multiple writes against the same item in the same transaction. For our application logic, however, It would be handy if it would be allowed to delete an item and insert a new version of it in the same transaction. Otherwise we would have to figure out in the app which deletes and adds are actually correlated and translate them into updates, which quickly gets messy, especially since we're using the mapper.

I've made an experimental change in TransactionItem::addRequestToMap which allows the insertion to go through if existingRequest instanceof Request.DeleteItem && (request instanceof Request.PutItem || request instanceof Request.UpdateItem). This seems to work fine.

What do you think? Is this a sufficient change for this feature and is it a good idea at all?

DynamoDB has transaction limit to write the items( = 100) in DDB atomically in a single transaction.

Hi,
DynamoDB native TX has the limit of 100 items(earlier it was 25) for transactionWrite in a single atomic transaction. So, does this library solves that problem because I have a use-case which will need more number of items(in multiples of 100) updates or save in a single DDB transaction?
Thanks
Aakash

Does this library support DAX ?

TransactionAssertionException exception without any description

com.amazonaws.services.dynamodbv2.transactions.exceptions.TransactionAssertionException: Item should not have been applied. Unable to release lock - item - {},

I recently started getting the TransactionAssertionException and I didn't find any documentation when do we receive this error? Can some please let me know if this is transient or some permanent
exception.

Plans for node.js sdk?

Hi, do you have plans for node.js SDK porting of this library?

Make compatible with SDK version 1.7.0

There have been some interface changes and a new version of Jackson that it depends on.

Cannot put a binary attribute in a transaction.

A json parsing exception is thrown when I add a binary attribute to a put request in a transaction.

Updating to support 1.7.7+

Someone else has already created this issue, however there is in response (from may last year).

Is this currently planning to be updated?

The Java SDK has added a fair few features (that make dynamodb specifically worthwhile when comparing them to other databases), such as documents and other types, all which are unusable

Transaction Record Data showing up in Main record Table

In MyTable I some times get records that have my hash key with the following fields
_TxD, _TxId, and _TxT and none of my fields aside from the hash key.

In the Transactions Table I can take the _TxId value from MyTable and look up a record there that has these fields
_TxId, _TxD, _TxR, _TxS, and _TxV

The TxD field on MyTable and in the Transaction Table do not have the same value (about 7 seconds difference) and the values of the other fields are provided below except _TxR

MyTable

1. _TxD value: 1505692898.128
2. _TxT value: 1

Transactions Table

1. _TxD value: 1505692981.343
2. _TxS value: R
3. _TxV value: 3
4. _TxR value: I suspect this to be an encrypted String that is all my requests involved in the transaction.

In what valid use cases would these transaction attribute values be written to MyTable instead of the data in the request involved in the transaction? Is there a way I can detect when the transaction library does this if it is a valid use case? If it is a valid use case what recovery steps should I take to re-process the transaction so my data is persisted into MyTable?

rollback on validation error

When I create a transaction that inserts a valid object followed by an invalid object that doesn't have a hash key nor a range key an error happens as expected, but I can't rollback, leaving the valid object inserted into the DB.

I understand that people should be sending valid objects, but I found this while trying learn the library by writing tests

Reading and then writting to the same item in a transaction cause error

1. Add a GetItemRequest for an item to the transaction
2. Add an UpdateItemRequest for the same item to the transaction
3. Commit the transaction
4. An error occurred: "Item should not have been applied. Unable to release lock"

After adding the UpdateItemRequest, the item is applied. When the transaction is committed, it loops through its requests to release locks. The GetItemRequest is added first, so it is handled first. The transaction will fail to release read-lock for the GetItemRequest because the item has been applied by the UpdateItemRequest.

A possible fix for this is that the transaction should not expect the item to be not applied when releasing read-lock.

Plans for Ruby SDK support?

Do you have plans to port this library for the AWS Ruby SDK?

Migration to aws-java-sdk 1.7.7

Is it possible to insert an item only if the hash+range key does not already exist?

When I add the withExpected conditions to the request, I get an exception. I considered running a transactional query and if it returns null, follow it with an insert. However, without a full table lock, two users could near-simultaneously insert the same hash+range, overwriting the previous item's attributes.

code:
t1.putItem(new PutItemRequest()
.withTableName("data")
.withItem(item)
.withExpected(expected));

Exception in thread "main" com.amazonaws.services.dynamodbv2.transactions.exceptions.InvalidRequestException: : Requests with conditions are not currently supported for transaction 454ea707-e332-4085-b8e0-7f4a5bc0f272 table data key {Value={S: [email protected],}, FieldPath={S: 37evEvXGWuegss5bBJEe2k.User._Username,}} -
at com.amazonaws.services.dynamodbv2.transactions.Request$PutItem.doValidate(Request.java:277)
at com.amazonaws.services.dynamodbv2.transactions.Request.validate(Request.java:298)
at com.amazonaws.services.dynamodbv2.transactions.Transaction.driveRequest(Transaction.java:512)
at com.amazonaws.services.dynamodbv2.transactions.Transaction.putItem(Transaction.java:169)
at com.salesfront.data.Database.saveUser(Database.java:113)
at com.salesfront.test.DraftTests.(DraftTests.java:49)
at com.salesfront.test.DraftTests.main(DraftTests.java:31)

Missing the key in the item when calling transaction.save()

I am trying to save a Item in DynamoDB using Lambda. If i replace the code with DynamoDBMapper. It works perfectly. I am using @DynamoDBAutoGeneratedKey annotation to generate the Id. Its not working even if i send the integration_endpoint_id explicitly.

Transaction transaction = txManager.newTransaction();
transaction.save(integrationEndpoint);
transaction.commit();

stacktrace:

{
  "errorMessage": "One or more parameter values were invalid: Missing the key integration_endpoint_id in the item (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: KG2DI2R78HR8ULKFSLPE36LJ5FVV4KQNSO5AEMVJF66Q9ASUAAJG)",
  "errorType": "com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException",
  "stackTrace": [
    "com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1387)",
    "com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:940)",
    "com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:715)",
    "com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:466)",
    "com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:427)",
    "com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:376)",
    "com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2078)",
    "com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2048)",
    "com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.putItem(AmazonDynamoDBClient.java:1584)",
    "com.amazonaws.services.dynamodbv2.transactions.TransactionItem.insert(TransactionItem.java:168)",
    "com.amazonaws.services.dynamodbv2.transactions.TransactionItem.<init>(TransactionItem.java:111)",
    "com.amazonaws.services.dynamodbv2.transactions.TransactionItem.<init>(TransactionItem.java:81)",
    "com.amazonaws.services.dynamodbv2.transactions.Transaction.<init>(Transaction.java:144)",
    "com.amazonaws.services.dynamodbv2.transactions.TransactionManager.newTransaction(TransactionManager.java:140)",

Is this project still active?

We were interested in using this but there are a few concerns/questions we have:

1. Is this project still active (there are a couple of PR's that are a few months old and some issues as old as 2015 that are unaddressed). Also no major commits in almost a year and a half.
2. Why isn't this published to Maven central to make dependency management more simple than shipping around JARs or building from source.
3. This looked promising but it's from 2013 https://aws.amazon.com/blogs/aws/dynamodb-transaction-library/ I haven't come across anything yet that indicates this library has been officially deprecated but it doesn't look to be particularly active or showing signs of having a roadmap.

Appreciate the work that's gone into this, just would be nice to have a clearer picture to help us assess whether it's appropriate for us moving forwards.

Thanks.

Support for requests with conditions

I would like to transfer money from bank account A to bank account B using dynamodb-transactions library.
So, I have to create 2 UpdateItemRequest.

1 - withdrawUpdateItemRequest
2 - donateUpdateItemRequest

For withdrawUpdateItemRequest I have to check that bank account really has such amount of money, so I need to use ExpectedAttributeValue map, but currently library does not support requests with conditions..

Do you have a plan to implement it? Could you tell any estimation for this issue?

boto3 python version?

Hi

Any chances that there will be a boto3 python version?

Could not run the example

Hello, thank you for this library!

I am currently porting it to nodejs, so I am diving into the details and it would be extremely useful if I could point the example class to a local instance of DynamoDB.

I've followed the instructions but I'm stuck at point 4.iv of Getting Started, this is the error I'm getting:

Downloading: http://repo1.maven.org/maven2/com/amazonaws/services/dynamodbv2/amazon-dynamodb-transactions/1.1.0/amazon-dynamodb-transact
ions-1.1.0.pom
[INFO] Unable to find resource 'com.amazonaws.services.dynamodbv2:amazon-dynamodb-transactions:pom:1.1.0' in repository central (http://
repo1.maven.org/maven2)
Downloading: http://repo1.maven.org/maven2/com/amazonaws/services/dynamodbv2/amazon-dynamodb-transactions/1.1.0/amazon-dynamodb-transact
ions-1.1.0.jar
[INFO] Unable to find resource 'com.amazonaws.services.dynamodbv2:amazon-dynamodb-transactions:jar:1.1.0' in repository central (http://
repo1.maven.org/maven2)
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.

Should I manually add the .jar to the project? Am I missing something obvious here?
Any hint would be appreciated!

Cheers

Alberto

Working with DynamoDBMapper

In order to use transactions we need to instantiate PutItemRequest and configure all attributes.
Is there any way to use transactions with DynamoDBMapper?

dynamodb transaction Manager under Javascript SDK

Hi, I would like to know if already exist TransactionManager for dynamodb under aws-sdk for JavaScript?
Thanks

dynamodb-transactions/examples/src/main/java/com/amazonaws/services/dynamodbv2/transactions/examples/TransactionExamples.java

Line 103 in a3e9f25

txManager = new TransactionManager(dynamodb, TX_TABLE_NAME, TX_IMAGES_TABLE_NAME);