oharaandrew314 / dynamodb-kotlin-module Goto Github PK
View Code? Open in Web Editor NEWKotlin Module for the dynamodb-enhanced SDk
License: Apache License 2.0
Kotlin Module for the dynamodb-enhanced SDk
License: Apache License 2.0
I see there is no Kt annotation for DynamoDbVersionAttribute. Will you be able to add it ? Or any workarounds that can be applied to get this functionality?
If there's a class with a nullable field, but no default value for it:
data class Person(val name: String, val age: Int?)
then the dynamo mapper will prefer to omit the null
value. When the mapper reads the item, object initialization fails because no value was provided for that null field.
The mapper should instead assume that a nullable field's default value is null
, unless otherwise indicated.
This backend is lighter and faster that kotlin-reflect, so it should reduce the footprint of this module.
I have the following requirement which is currently not working as expected.
There is a CustomDataClass
that has the property of a custom type Subtyp
for which a converter exists. Inside the Subtyp
there is a property of the Map<String, Any>
type.
Even if there is a converter for the Subtyp
class that is handling the transformation it is necessary to add an additional converter to the Map<String, Any>
property in order to make the database creation call successfully.
See below test case for the described problem:
@Autowired private val enhancedAsyncClient: DynamoDbEnhancedAsyncClient,
) {
data class Subtyp(
val map: Map<String, Any>
)
class SubtypeConverter: AttributeConverter<Subtyp>{
override fun transformFrom(input: Subtyp?): AttributeValue {
TODO("Not yet implemented")
}
override fun transformTo(input: AttributeValue?): Subtyp {
TODO("Not yet implemented")
}
override fun type(): EnhancedType<Subtyp> {
TODO("Not yet implemented")
}
override fun attributeValueType(): AttributeValueType {
TODO("Not yet implemented")
}
}
@DynamoDbBean
data class CustomDataClass(
@DynamoKtPartitionKey
val id: String,
@DynamoKtConverted(SubtypeConverter::class)
val subtyp: Subtyp
)
@Test
fun `properties of Subtyp for which a converter exist should not be scanned for a converter` () = runBlockingJUnit{
enhancedAsyncClient
.table(UUID.randomUUID().toString(), DataClassTableSchema(CustomDataClass::class))
.also { it.createTable().await() }
// This leads to
// java.lang.IndexOutOfBoundsException: Index: 0
// at java.base/java.util.Collections$EmptyList.get(Collections.java:4586)
// at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.createMapConverter(DefaultAttributeConverterProvider.java:191)
// at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.findConverterInternal(DefaultAttributeConverterProvider.java:161)
// at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.findConverter(DefaultAttributeConverterProvider.java:145)
// at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.createMapConverter(DefaultAttributeConverterProvider.java:195)
// at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.findConverterInternal(DefaultAttributeConverterProvider.java:161)
// at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.findConverter(DefaultAttributeConverterProvider.java:145)
// at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.converterFor(DefaultAttributeConverterProvider.java:137)
// at io.andrewohara.dynamokt.ImmutableDataClassAttributeKt.toImmutableDataClassAttribute(ImmutableDataClassAttribute.kt:86)
// at io.andrewohara.dynamokt.DataClassTableSchemaKt.dataClassTableSchema(DataClassTableSchema.kt:43)
// at io.andrewohara.dynamokt.DataClassTableSchemaKt.recursiveDataClassTableSchema(DataClassTableSchema.kt:60)
// at io.andrewohara.dynamokt.ImmutableDataClassAttributeKt.toEnhancedType(ImmutableDataClassAttribute.kt:35)
// at io.andrewohara.dynamokt.ImmutableDataClassAttributeKt.toImmutableDataClassAttribute(ImmutableDataClassAttribute.kt:94)
// at io.andrewohara.dynamokt.DataClassTableSchemaKt.dataClassTableSchema(DataClassTableSchema.kt:43)
// at io.andrewohara.dynamokt.DataClassTableSchemaKt$DataClassTableSchema$1.invoke(DataClassTableSchema.kt:18)
// at io.andrewohara.dynamokt.DataClassTableSchemaKt$DataClassTableSchema$1.invoke(DataClassTableSchema.kt:17)
// at io.andrewohara.dynamokt.DataClassTableSchemaKt.DataClassTableSchema$lambda$0(DataClassTableSchema.kt:17)
// at java.base/java.util.Map.computeIfAbsent(Map.java:1054)
}
Is it possible to stop scanning for inner data types if the outer one already contains a converter?
When a mapping error occurs (e.g. when a required field is null as in this example) the current behaviour is not as expected:
updateItem
scan
The only method behaving as expected (throwing an exception) is the query-method.
Here is an example Code detailing the problem (make sure to use a test dynamodb since the test deletes all other entities before running)
import io.andrewohara.dynamokt.DataClassTableSchema
import io.andrewohara.dynamokt.DynamoKtPartitionKey
import io.kotest.matchers.shouldBe
import io.kotest.matchers.shouldNotBe
import kotlinx.coroutines.future.await
import org.junit.jupiter.api.Test
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.context.SpringBootTest
import org.springframework.test.context.ActiveProfiles
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedAsyncClient
import software.amazon.awssdk.enhanced.dynamodb.Key
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbBean
import software.amazon.awssdk.enhanced.dynamodb.model.GetItemEnhancedRequest
const val TABLE_NAME = "INSERT_NAME_OF_YOUR_TABLE_HERE"
@SpringBootTest
class ScanAllExample(
@Autowired private val enhancedClient: DynamoDbEnhancedAsyncClient,
) {
@Test
fun shouldSaveEntity() = runBlocking {
val testClient = enhancedClient.table(TABLE_NAME, DataClassTableSchema(TestEntityV1::class))
enhancedClient.table(TABLE_NAME, DataClassTableSchema(TestEntityV2::class))
//delete item if already exists
testClient.deleteItem(Key.builder().partitionValue("test-id").build())
//save item with id 'test-id'
testClient.updateItem(TestEntityV1(id = "test-id", name = null)).await()
//get an item with 'test-id'
val item = testClient.getItem(
GetItemEnhancedRequest.builder()
.consistentRead(true)
.key(Key.builder().partitionValue("test-id").build())
.build(),
).await()
//item was not found in db (actually it was never stored - even though no exception was thrown)
item shouldNotBe null
}
@Test
fun shouldBehaveCorrectlyWithScanAll() = runBlocking {
val legacyClient = enhancedClient.table(TABLE_NAME, DataClassTableSchema(TestEntityV1::class))
legacyClient.scan().subscribe { list ->
list.items().forEach {
println("Deleting ${it.id}")
legacyClient.deleteItem(Key.builder().partitionValue(it.id).build())
}
}.get()
val valid = legacyClient.updateItem(TestEntityV1(id = "valid", name = "valid name")).await()
val invalid = legacyClient.updateItem(TestEntityV1(id = "invalid", name = null)).await()
valid shouldNotBe null
invalid shouldNotBe null
legacyClient.scan().subscribe { items ->
items.items().forEach {
println("Legacy Client found ${it.id} in database")
}
}.get()
val newClient = enhancedClient.table(TABLE_NAME, DataClassTableSchema(TestEntityV2::class))
val setOfFound = mutableSetOf<String>()
println("Scanning repository")
newClient.scan().subscribe { items ->
items.items().forEach {
println("New Client Found ${it.id} in database")
setOfFound.add(it.id)
}
}.get()
//the code never reaches this point
setOfFound.size shouldBe 2
setOfFound.contains("valid") shouldBe true
setOfFound.contains("invalid") shouldBe true
}
}
@DynamoDbBean()
data class TestEntityV2(
@DynamoKtPartitionKey
var id: String,
val name: String,
val otherField: String = "abc"
)
@DynamoDbBean()
data class TestEntityV1(
@DynamoKtPartitionKey
var id: String,
val name: String?,
val otherField: String = "abc"
)
When using createTableWithIndices to create a table with indices (global but not local) I get the following error
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'tableInitializer': Invocation of init method failed; nested exception is software.amazon.awssdk.services.dynamodb.model.DynamoDbException: LSI list is empty/invalid (Service: DynamoDb, Status Code: 400, Request ID: 4c1beac5-c3cc-409e-89f9-778c1b2a27d0, Extended Request ID: null)
I believe this could be resolved like this below.
val builder = CreateTableEnhancedRequest.builder()
globalIndices.takeIf { it.isNotEmpty() }?.apply { builder.globalSecondaryIndices(this) }
localIndices.takeIf { it.isNotEmpty() }?.apply { builder.localSecondaryIndices(this) }
val request = builder
.build()
createTable(request)
note: I wasn't able to reproduce my issue with your local tests
Need to add support for classes with recursive properties, e.g.
data class Foo(val foo: Foo?)
I have data class with somer properties defined in constructor and some in class body, e.g.
data class MyClass(
@DynamoKtAttribute
var prop1: String? = null
) {
@DynamoKtAttribute
var prop2: String = "def"
}
When schema for it created, code here https://github.com/oharaandrew314/dynamodb-kotlin-module/blob/master/src/main/kotlin/io/andrewohara/dynamokt/DataClassAttribute.kt#L55
builds incorrect association between attributes and constructor parameter.
So later mapToItem fails to create instance because of constructor arguments mismatch.
If a property is annotated with a ttl annotation, then the table should be created with a ttl attribute
With version 0.4.0 the createTableWithIndices
functions were removed, because the underlying DefaultDynamoDbTable
already is doing the job.
This is working for the DefaultDynamoDbTable
but not for the DynamoDbEnhancedAsyncClient
as I can see. The tests in CreateTableTest
should verify the correct behavior after the removal of the createTableWithIndices
- function are not using the async variant.
After replacing the createTableWithIndices
- function calls with the recommended createTable
- function calls in my application tests are failing because of missing indices.
Below you can find an updated test that I use to reproduce the issue. It is showing me that there are no indices available.
class CreateTableTest {
private data class Person(
@DynamoKtPartitionKey val id: Int,
@DynamoKtSecondaryPartitionKey(indexNames = ["names"]) val name: String,
@DynamoKtSecondarySortKey(indexNames = ["names"]) val dob: Instant
)
private val storage = Storage.InMemory<DynamoTable>()
private val personTable = DynamoDbEnhancedAsyncClient.builder()
.dynamoDbClient(
DynamoDbAsyncClient.builder()
.httpClient(AwsSdkAsyncClient(FakeDynamoDb(storage)))
.credentialsProvider { AwsBasicCredentials.create("key", "id") }
.region(Region.CA_CENTRAL_1)
.build()
)
.build()
.table("people", DataClassTableSchema(Person::class))
@Test
fun createTable() = runBlocking {
personTable.createTable().await()
val table = storage["people"].shouldNotBeNull().table
table.GlobalSecondaryIndexes.shouldContainExactlyInAnyOrder(
GlobalSecondaryIndexResponse(
IndexName = "names",
KeySchema = listOf(
KeySchema(AttributeName.of("name"), KeyType.HASH),
KeySchema(AttributeName.of("dob"), KeyType.RANGE)
),
Projection = Projection(ProjectionType = ProjectionType.ALL)
)
)
table.LocalSecondaryIndexes.shouldBeNull()
}
}
Can you please verify if this is correct?
The "original createTable will lose indices" unit test will fail if dynamodb-enhanced
is upgraded to a newer 2.20.+ version.
> Task :test
CreateTableTest > original createTable will lose indices() FAILED
java.lang.AssertionError at CreateTableTest.kt:31
57 tests completed, 1 failed
The mapper cannot properly convert null lists or sets.
java.lang.IllegalStateException: Unable to convert attribute value: AttributeValue()
at software.amazon.awssdk.enhanced.dynamodb.internal.converter.attribute.EnhancedAttributeValue.fromAttributeValue(EnhancedAttributeValue.java:352)
at software.amazon.awssdk.enhanced.dynamodb.internal.converter.attribute.ListAttributeConverter$Delegate.transformTo(ListAttributeConverter.java:154)
at software.amazon.awssdk.enhanced.dynamodb.internal.converter.attribute.ListAttributeConverter.transformTo(ListAttributeConverter.java:120)
at software.amazon.awssdk.enhanced.dynamodb.internal.converter.attribute.ListAttributeConverter.transformTo(ListAttributeConverter.java:82)
at io.andrewohara.dynamokt.DataClassAttribute.unConvert(DataClassAttribute.kt:34)
at io.andrewohara.dynamokt.DataClassTableSchema.mapToItem(DataClassTableSchema.kt:21)
at software.amazon.awssdk.enhanced.dynamodb.TableSchema.mapToItem(TableSchema.java:185)
at software.amazon.awssdk.enhanced.dynamodb.internal.converter.attribute.DocumentAttributeConverter.transformTo(DocumentAttributeConverter.java:62)
at io.andrewohara.dynamokt.DataClassAttribute.unConvert(DataClassAttribute.kt:34)
at io.andrewohara.dynamokt.DataClassTableSchema.mapToItem(DataClassTableSchema.kt:21)
at software.amazon.awssdk.enhanced.dynamodb.internal.EnhancedClientUtils.readAndTransformSingleItem(EnhancedClientUtils.java:87)
at software.amazon.awssdk.enhanced.dynamodb.internal.operations.PutItemOperation.transformResponse(PutItemOperation.java:127)
at software.amazon.awssdk.enhanced.dynamodb.internal.operations.PutItemOperation.transformResponse(PutItemOperation.java:45)
at software.amazon.awssdk.enhanced.dynamodb.internal.operations.CommonOperation.execute(CommonOperation.java:115)
at software.amazon.awssdk.enhanced.dynamodb.internal.operations.TableOperation.executeOnPrimaryIndex(TableOperation.java:59)
at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbTable.putItem(DefaultDynamoDbTable.java:201)
at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbTable.putItem(DefaultDynamoDbTable.java:209)
at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbTable.putItem(DefaultDynamoDbTable.java:214)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.