oharaandrew314 / dynamodb-kotlin-module Goto Github PK

View Code? Open in Web Editor NEW

23.0 6.0 3.0 149 KB

Kotlin Module for the dynamodb-enhanced SDk

License: Apache License 2.0

Kotlin 100.00%

kotlin aws aws-sdk-java-v2 dynamodb mapper

dynamodb-kotlin-module's People

Contributors

Stargazers

Watchers

Forkers

sleungtoast cooco119 donarus

dynamodb-kotlin-module's Issues

Support DynamoDbPreserveEmptyObject annotation

Support for DynamoDbVersionAttribute to enable Optimistic Locking

I see there is no Kt annotation for DynamoDbVersionAttribute. Will you be able to add it ? Or any workarounds that can be applied to get this functionality?

Nullable field without default value cannot be converted back to object

If there's a class with a nullable field, but no default value for it:

data class Person(val name: String, val age: Int?)

then the dynamo mapper will prefer to omit the null value. When the mapper reads the item, object initialization fails because no value was provided for that null field.

The mapper should instead assume that a nullable field's default value is null, unless otherwise indicated.

Replace reflective backend with kotlinx-metadata-jvm

This backend is lighter and faster that kotlin-reflect, so it should reduce the footprint of this module.

Adding a custom converter to a data class property still makes additional converters for properties of the inner type necessary.

I have the following requirement which is currently not working as expected.

There is a CustomDataClass that has the property of a custom type Subtyp for which a converter exists. Inside the Subtyp there is a property of the Map<String, Any> type.
Even if there is a converter for the Subtyp class that is handling the transformation it is necessary to add an additional converter to the Map<String, Any> property in order to make the database creation call successfully.

See below test case for the described problem:

    @Autowired private val enhancedAsyncClient: DynamoDbEnhancedAsyncClient,
) {

    data class Subtyp(
        val map: Map<String, Any>
    )

    class SubtypeConverter: AttributeConverter<Subtyp>{
        override fun transformFrom(input: Subtyp?): AttributeValue {
            TODO("Not yet implemented")
        }

        override fun transformTo(input: AttributeValue?): Subtyp {
            TODO("Not yet implemented")
        }

        override fun type(): EnhancedType<Subtyp> {
            TODO("Not yet implemented")
        }

        override fun attributeValueType(): AttributeValueType {
            TODO("Not yet implemented")
        }

    }

    @DynamoDbBean
    data class CustomDataClass(
        @DynamoKtPartitionKey
        val id: String,
        @DynamoKtConverted(SubtypeConverter::class)
        val subtyp: Subtyp
    )


    @Test
    fun `properties of Subtyp for which a converter exist should not be scanned for a converter` () = runBlockingJUnit{

        enhancedAsyncClient
            .table(UUID.randomUUID().toString(), DataClassTableSchema(CustomDataClass::class))
            .also { it.createTable().await() }

        // This leads to
        //        java.lang.IndexOutOfBoundsException: Index: 0
        //        at java.base/java.util.Collections$EmptyList.get(Collections.java:4586)
        //        at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.createMapConverter(DefaultAttributeConverterProvider.java:191)
        //        at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.findConverterInternal(DefaultAttributeConverterProvider.java:161)
        //        at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.findConverter(DefaultAttributeConverterProvider.java:145)
        //        at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.createMapConverter(DefaultAttributeConverterProvider.java:195)
        //        at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.findConverterInternal(DefaultAttributeConverterProvider.java:161)
        //        at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.findConverter(DefaultAttributeConverterProvider.java:145)
        //        at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.converterFor(DefaultAttributeConverterProvider.java:137)
        //        at io.andrewohara.dynamokt.ImmutableDataClassAttributeKt.toImmutableDataClassAttribute(ImmutableDataClassAttribute.kt:86)
        //        at io.andrewohara.dynamokt.DataClassTableSchemaKt.dataClassTableSchema(DataClassTableSchema.kt:43)
        //        at io.andrewohara.dynamokt.DataClassTableSchemaKt.recursiveDataClassTableSchema(DataClassTableSchema.kt:60)
        //        at io.andrewohara.dynamokt.ImmutableDataClassAttributeKt.toEnhancedType(ImmutableDataClassAttribute.kt:35)
        //        at io.andrewohara.dynamokt.ImmutableDataClassAttributeKt.toImmutableDataClassAttribute(ImmutableDataClassAttribute.kt:94)
        //        at io.andrewohara.dynamokt.DataClassTableSchemaKt.dataClassTableSchema(DataClassTableSchema.kt:43)
        //        at io.andrewohara.dynamokt.DataClassTableSchemaKt$DataClassTableSchema$1.invoke(DataClassTableSchema.kt:18)
        //        at io.andrewohara.dynamokt.DataClassTableSchemaKt$DataClassTableSchema$1.invoke(DataClassTableSchema.kt:17)
        //        at io.andrewohara.dynamokt.DataClassTableSchemaKt.DataClassTableSchema$lambda$0(DataClassTableSchema.kt:17)
        //        at java.base/java.util.Map.computeIfAbsent(Map.java:1054)
    }

Is it possible to stop scanning for inner data types if the outer one already contains a converter?

Save fails without exception and scan runs forever when a mapping error occurs

When a mapping error occurs (e.g. when a required field is null as in this example) the current behaviour is not as expected:

updateItem

current behaviour: Executes without an error so that it seems like everything worked but the entity is not persisted into DynamoDB
expected behavior: An exception is thrown detailing the error

scan

current behaviour: the scan never finishes, running forever
expected behaviour: An exception is thrown on the defect entity detailling the mapping error which occurred

The only method behaving as expected (throwing an exception) is the query-method.

Here is an example Code detailing the problem (make sure to use a test dynamodb since the test deletes all other entities before running)

import io.andrewohara.dynamokt.DataClassTableSchema
import io.andrewohara.dynamokt.DynamoKtPartitionKey
import io.kotest.matchers.shouldBe
import io.kotest.matchers.shouldNotBe
import kotlinx.coroutines.future.await
import org.junit.jupiter.api.Test
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.context.SpringBootTest
import org.springframework.test.context.ActiveProfiles
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedAsyncClient
import software.amazon.awssdk.enhanced.dynamodb.Key
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbBean
import software.amazon.awssdk.enhanced.dynamodb.model.GetItemEnhancedRequest

const val TABLE_NAME = "INSERT_NAME_OF_YOUR_TABLE_HERE"

@SpringBootTest
class ScanAllExample(
    @Autowired private val enhancedClient: DynamoDbEnhancedAsyncClient,
) {

    @Test
    fun shouldSaveEntity() = runBlocking {
        val testClient = enhancedClient.table(TABLE_NAME, DataClassTableSchema(TestEntityV1::class))
        enhancedClient.table(TABLE_NAME, DataClassTableSchema(TestEntityV2::class))
        //delete item if already exists
        testClient.deleteItem(Key.builder().partitionValue("test-id").build())
        //save item with id 'test-id'
        testClient.updateItem(TestEntityV1(id = "test-id", name = null)).await()
        //get an item with 'test-id'
        val item = testClient.getItem(
            GetItemEnhancedRequest.builder()
                .consistentRead(true)
                .key(Key.builder().partitionValue("test-id").build())
                .build(),
        ).await()
        //item was not found in db (actually it was never stored - even though no exception was thrown)
        item shouldNotBe null
    }

    @Test
    fun shouldBehaveCorrectlyWithScanAll() = runBlocking {
        val legacyClient = enhancedClient.table(TABLE_NAME, DataClassTableSchema(TestEntityV1::class))
        legacyClient.scan().subscribe { list ->
            list.items().forEach {
                println("Deleting ${it.id}")
                legacyClient.deleteItem(Key.builder().partitionValue(it.id).build())
            }
        }.get()
        val valid = legacyClient.updateItem(TestEntityV1(id = "valid", name = "valid name")).await()
        val invalid = legacyClient.updateItem(TestEntityV1(id = "invalid", name = null)).await()
        valid shouldNotBe null
        invalid shouldNotBe null

        legacyClient.scan().subscribe { items ->
            items.items().forEach {
                println("Legacy Client found ${it.id} in database")
            }
        }.get()

        val newClient = enhancedClient.table(TABLE_NAME, DataClassTableSchema(TestEntityV2::class))
        
        val setOfFound = mutableSetOf<String>()
        println("Scanning repository")
        newClient.scan().subscribe { items ->
            items.items().forEach {
                println("New Client Found ${it.id} in database")
                setOfFound.add(it.id)
            }
        }.get()

        //the code never reaches this point

        setOfFound.size shouldBe 2
        setOfFound.contains("valid") shouldBe true
        setOfFound.contains("invalid") shouldBe true
    }
}

@DynamoDbBean()
data class TestEntityV2(
    @DynamoKtPartitionKey
    var id: String,
    val name: String,
    val otherField: String = "abc"
)

@DynamoDbBean()
data class TestEntityV1(
    @DynamoKtPartitionKey
    var id: String,
    val name: String?,
    val otherField: String = "abc"
)

empty Local secondary indices results in error when calling createTableWithIndices

When using createTableWithIndices to create a table with indices (global but not local) I get the following error
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'tableInitializer': Invocation of init method failed; nested exception is software.amazon.awssdk.services.dynamodb.model.DynamoDbException: LSI list is empty/invalid (Service: DynamoDb, Status Code: 400, Request ID: 4c1beac5-c3cc-409e-89f9-778c1b2a27d0, Extended Request ID: null)

I believe this could be resolved like this below.

val builder = CreateTableEnhancedRequest.builder()
    globalIndices.takeIf { it.isNotEmpty() }?.apply { builder.globalSecondaryIndices(this) }
    localIndices.takeIf { it.isNotEmpty() }?.apply { builder.localSecondaryIndices(this) }
    val request = builder
        .build()

    createTable(request)

note: I wasn't able to reproduce my issue with your local tests

Support Recursive class structure

Need to add support for classes with recursive properties, e.g.

data class Foo(val foo: Foo?)

Doesn't support properties defined in class body

I have data class with somer properties defined in constructor and some in class body, e.g.

data class MyClass(
    @DynamoKtAttribute
    var prop1: String? = null
) {
    @DynamoKtAttribute
    var prop2: String = "def"  
}

When schema for it created, code here https://github.com/oharaandrew314/dynamodb-kotlin-module/blob/master/src/main/kotlin/io/andrewohara/dynamokt/DataClassAttribute.kt#L55
builds incorrect association between attributes and constructor parameter.

So later mapToItem fails to create instance because of constructor arguments mismatch.

Support TTL annotation on property

If a property is annotated with a ttl annotation, then the table should be created with a ttl attribute

Add support for DynamoDbFlatten

Functionality of createTable changed from version 0.4.0 on for DynamoDbEnhancedAsyncClient

With version 0.4.0 the createTableWithIndices functions were removed, because the underlying DefaultDynamoDbTable already is doing the job.

This is working for the DefaultDynamoDbTable but not for the DynamoDbEnhancedAsyncClient as I can see. The tests in CreateTableTest should verify the correct behavior after the removal of the createTableWithIndices - function are not using the async variant.

After replacing the createTableWithIndices - function calls with the recommended createTable - function calls in my application tests are failing because of missing indices.

Below you can find an updated test that I use to reproduce the issue. It is showing me that there are no indices available.

class CreateTableTest {

    private data class Person(
        @DynamoKtPartitionKey val id: Int,
        @DynamoKtSecondaryPartitionKey(indexNames = ["names"]) val name: String,
        @DynamoKtSecondarySortKey(indexNames = ["names"]) val dob: Instant
    )

    private val storage = Storage.InMemory<DynamoTable>()

    private val personTable = DynamoDbEnhancedAsyncClient.builder()
        .dynamoDbClient(
            DynamoDbAsyncClient.builder()
                .httpClient(AwsSdkAsyncClient(FakeDynamoDb(storage)))
                .credentialsProvider { AwsBasicCredentials.create("key", "id") }
                .region(Region.CA_CENTRAL_1)
                .build()
        )
        .build()
        .table("people", DataClassTableSchema(Person::class))


    @Test
    fun createTable() = runBlocking {
        personTable.createTable().await()

        val table = storage["people"].shouldNotBeNull().table

        table.GlobalSecondaryIndexes.shouldContainExactlyInAnyOrder(
            GlobalSecondaryIndexResponse(
                IndexName = "names",
                KeySchema = listOf(
                    KeySchema(AttributeName.of("name"), KeyType.HASH),
                    KeySchema(AttributeName.of("dob"), KeyType.RANGE)
                ),
                Projection = Projection(ProjectionType = ProjectionType.ALL)
            )
        )

        table.LocalSecondaryIndexes.shouldBeNull()
    }
}

Can you please verify if this is correct?

CreateTableTest fails with newer dynamodb-enhanced:2.20.+ version

The "original createTable will lose indices" unit test will fail if dynamodb-enhanced is upgraded to a newer 2.20.+ version.

> Task :test

CreateTableTest > original createTable will lose indices() FAILED
    java.lang.AssertionError at CreateTableTest.kt:31

57 tests completed, 1 failed

Support Value Objects

Support null sets and lists

The mapper cannot properly convert null lists or sets.

java.lang.IllegalStateException: Unable to convert attribute value: AttributeValue()
	at software.amazon.awssdk.enhanced.dynamodb.internal.converter.attribute.EnhancedAttributeValue.fromAttributeValue(EnhancedAttributeValue.java:352)
	at software.amazon.awssdk.enhanced.dynamodb.internal.converter.attribute.ListAttributeConverter$Delegate.transformTo(ListAttributeConverter.java:154)
	at software.amazon.awssdk.enhanced.dynamodb.internal.converter.attribute.ListAttributeConverter.transformTo(ListAttributeConverter.java:120)
	at software.amazon.awssdk.enhanced.dynamodb.internal.converter.attribute.ListAttributeConverter.transformTo(ListAttributeConverter.java:82)
	at io.andrewohara.dynamokt.DataClassAttribute.unConvert(DataClassAttribute.kt:34)
	at io.andrewohara.dynamokt.DataClassTableSchema.mapToItem(DataClassTableSchema.kt:21)
	at software.amazon.awssdk.enhanced.dynamodb.TableSchema.mapToItem(TableSchema.java:185)
	at software.amazon.awssdk.enhanced.dynamodb.internal.converter.attribute.DocumentAttributeConverter.transformTo(DocumentAttributeConverter.java:62)
	at io.andrewohara.dynamokt.DataClassAttribute.unConvert(DataClassAttribute.kt:34)
	at io.andrewohara.dynamokt.DataClassTableSchema.mapToItem(DataClassTableSchema.kt:21)
	at software.amazon.awssdk.enhanced.dynamodb.internal.EnhancedClientUtils.readAndTransformSingleItem(EnhancedClientUtils.java:87)
	at software.amazon.awssdk.enhanced.dynamodb.internal.operations.PutItemOperation.transformResponse(PutItemOperation.java:127)
	at software.amazon.awssdk.enhanced.dynamodb.internal.operations.PutItemOperation.transformResponse(PutItemOperation.java:45)
	at software.amazon.awssdk.enhanced.dynamodb.internal.operations.CommonOperation.execute(CommonOperation.java:115)
	at software.amazon.awssdk.enhanced.dynamodb.internal.operations.TableOperation.executeOnPrimaryIndex(TableOperation.java:59)
	at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbTable.putItem(DefaultDynamoDbTable.java:201)
	at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbTable.putItem(DefaultDynamoDbTable.java:209)
	at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbTable.putItem(DefaultDynamoDbTable.java:214)