n3twork / dynamap Goto Github PK
View Code? Open in Web Editor NEWDynamap is Java object model and mapping library for Amazon's DynamoDB database.
Home Page: https://dynamap.n3twork.com
License: Apache License 2.0
Dynamap is Java object model and mapping library for Amazon's DynamoDB database.
Home Page: https://dynamap.n3twork.com
License: Apache License 2.0
API is:
With only HasKey
scanspec.withExclusiveStartKey(String hashKeyName, Object hashKeyValue)
With HashKey and RangeKey
scanspec.withExclusiveStartKey(String hashKeyName, Object hashKeyValue, String rangeKeyName, Object rangeKeyValue)
Version: 0.9.56
After the generation of the Dynmap class files, Eclipse reports errors for each interface constant used in the @JsonProperty
annotation used in the bean class.
Referencing the interface in the annotation resolves this, e.g. @JsonProperty(Application.ID_FIELD)
{
"tables": [
{
"table": "Application",
"package": "com.example.application",
"type": "Application",
"version": 1,
"hashKey": "id",
"types": [
{
"name": "Application",
"description": "New application",
"fields": [
{
"name": "id",
"description": "Primary key",
"dynamoName": "Id",
"type": "String"
},
{
"name": "profile",
"description": "Profile name",
"dynamoName": "profile",
"type": "String"
},
{
"name": "lastUpdatedAt",
"description": "Updated date/time",
"dynamoName": "LastUpdatedAt",
"type": "String"
},
{
"name": "applicationName",
"description": "The application name",
"dynamoName": "ApplicationName",
"type": "String"
}
]
}
]
}
]
}
For various reasons, we want a workflow where we can run the Dynamap code generator from the commandline or called from a driver script. Currently, this is difficult because we need to build a classpath for the Dynamap code generator and that involves tracking down many jar files.
Ideally, the mvn artifact for Dynamap would include an application jar version of the code generator so we could simply:
java -jar dynamapcodegen.jar <args>
This would make it easy to integrate into the workflow we're adopting.
thanks!
Reference Quickstart guide, Step 2. Define a schema.
The following updates are required:
{
"tables": [
{
"table": "User",
"package": "com.n3twork.dynamap.example",
"type": "User",
"version": 1,
"hashKey": "id",
...
}]
}
When a table is configured with a GSI with projection type KEYS_ONLY, the GSI is created with projection type ALL instead.
Bad schema:
{
"tables": [
{
"table": "Test",
"package": "com.n3twork.wom.test",
"schemaVersionField": "schemaVersion",
"type": "Test",
"version": 1,
"hashKey": "id",
"optimisticLocking": true,
"types": [
{
"name": "Test",
"fields": [
{
"name": "id",
"dynamoName": "id",
"type": "String"
},
{
"name": "revision",
"dynamoName": "revision",
"type": "String"
}
]
}
]
}
]
}
Conditional updates that use the NOT_EQUALS operato fails with an exception like:
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Invalid ConditionExpression: Syntax error; token: "!", near: "#t1a0 !=" (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: b1d63797-a48d-4c35-8a7c-45c92c508b08)
This is "<>" should be used instead of "!=" to represent "not-equals" in the update condition according to:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.OperatorsAndFunctions.html
Specifying values in equalsFields
in a Dynamap schema that are not real fields defined in the schema should be a codegen error... but currently Dynamap will include the fields in the generated equals
method which results in broken code.
Take for example this (partial) schema, where the root type has a field that’s a list of element of another generated type:
{ …, "types": [
{
"name": "Root",
"description": "The table’s root type",
"fields": [
{ "name": "id", "dynamoName": "id", "description": "The hash key", "type": "String" },
{ "name": "listOfGenerated", "dynamoName": "logen", "type": "List", "elementType": "GeneratedListElement" }
]
},
{
"name": "GeneratedListElement",
"fields": [
{ "name": "foo", "dynamoName": "foo", "type": "String" }
]
}
] }
The generated RootBean
class will have the following annotated member variable, where the argument to the annotation contains invalid syntax:
@JsonDeserialize(as=List<GeneratedListElement>Bean.class)
private List< GeneratedListElement> listOfGenerated;
This invalid code is generated by line 42 of bean.ftl (as of current master, 952ed0e):
@JsonDeserialize(as=<@field_type field=field />Bean.class)
When using optimistic locking, the bean return by the migration does not include the updated revision number.
If you then try to do an update using an updates object created from this bean then it will fail due to condition check.
Ran into this refactoring a schema that was previously persist: false
, so it didn't need a dynamoName
. It will now be persisted and I failed to provide a dynamoName
. The error I get is pretty unhelpful. I think we can add this to a more general "validation" task for schemas.
Exception in thread "main" java.lang.RuntimeException: Error processing ./schemas/guild.json
at com.n3twork.dynamap.CodeGenerator.main(CodeGenerator.java:64)
Caused by: java.lang.IllegalArgumentException: Type: Guild has a duplicated field dynamo name: null. Field dynamo name must be unique.
at com.n3twork.dynamap.CodeGenerator.generateSchemaClasses(CodeGenerator.java:115)
at com.n3twork.dynamap.CodeGenerator.generateCode(CodeGenerator.java:80)
at com.n3twork.dynamap.CodeGenerator.main(CodeGenerator.java:62)
Generated interface fields are in the format without visibility, scope and final modifiers, i.e. String ID_FIELD = "Id";
The variables should have the public static final
to be true constants, i.e.
public static final String ID_FIELD = "Id";
Dynamap attempts to migrate query results for GSIs that are not configured with ALL projection type, which may result in the log message: "Schema version field does not exist for {} on item with hash key {}. Migrating item to current version"
In this case the message is triggered because the schema version field is not being projected into the index.
The "map" that I was adding addCheckAttributeInMapNotExistsCondition and addCheckAttributeInMapExistsCondition on was defined like this:
"compressCollection": "gzip"
So obviously that isn't going to work! The scary thing is that a addCheckAttributeInMapNotExistsCondition on a gzipped map passes. The addCheckAttributeInMapExistsCondition was failing. Dynamap boils these down to simple attribute_exists() statements in DynamoDB... so it makes sense that this is the behavior we get.
This could be dangerous though for uses of addCheckAttributeInMapNotExistsCondition. I just checked the few usages on our game and they look okay but I can easily imagine such a bug being introduced when we scramble to save document space and migrate to gzipped maps without noticing that we neutered one of these conditional checks.
I don't think there's a straightforward fix in Dynamap... we would have to find a way to vary the DynamapExpressionBuilder based on the Bean type. Not impossible... but probably something that needs careful consideration.
The schema should fail validation.
Dynamap will create tables in DynamoDB if they don't yet exist. It creates them with explicit provisioning and 1 read + write IOP. Better to just create the table with on demand provisioning... that's likely to be how the table stays for dev and live environments.
//FIXME:dynamap should ensure we always get a list. On version 0.9.25 it may return null if none was in storage
List list = dynamap.batchGetObjectSingleCollection(new BatchGetObjectRequest()
.withGetObjectRequests(beanRequests));
return list != null ? list : Collections.emptyList();
Currently it is not possible to replace an entire nested generated type object in a parent updates object instead of using it's associated updates object if the nested type contains compressed or serialize as list fields.
Note, however, in most situations it should not be necessary to replace a nested generated object like this. The whole point of using generated types is that you can work with an updates object. This should even work for creating brand new objects.
Ran into this just now: have a schema that lists specific equalsFields
and I dropped one of those fields from the core schema.
Expected behavior: Schema validation error, invalid equalsFields
.
Actual behavior: Code generation succeeds and uses the invalid fields listed in equalsFields
leading to invalid Java code.
Given a field with this configuration:
{
"name": "stringList",
"dynamoName": "stringList",
"type": "List",
"elementType": "String"
}
The generated setStringList
method will set the in-memory state of the *Updates
object, but the list is not actually persisted since only the list adds are used when generating the Dynamo DB update expression. On the other hand, setting deltas: false
will cause values passed to setStringList
to be persisted. It may be less confusing if setStringList
is not generated at all in the case of deltas: true
.
This can cause Jackson to attempt to deserialize such fields out of the field loaded from Dynamo DB which may result in exceptions like the following:
! java.lang.IllegalArgumentException: Conflicting setter definitions for property "consumedCapacity": com.n3twork.dynamap.DynamoRateLimiter#setConsumedCapacity(java.util.List) vs com.n3twork.dynamap.DynamoRateLimiter#setConsumedCapacity(com.amazonaws.services.dynamodbv2.model.ConsumedCapacity)
! at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.getSetter(POJOPropertyBuilder.java:512)
! at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.getPrimaryType(POJOPropertyBuilder.java:353)
! at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.getRawPrimaryType(POJOPropertyBuilder.java:373)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.filterBeanProps(BeanDeserializerFactory.java:709)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.addBeanProps(BeanDeserializerFactory.java:559)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:277)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:150)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:414)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
! ... 84 common frames omitted
! Causing: com.fasterxml.jackson.databind.JsonMappingException: Conflicting setter definitions for property "consumedCapacity": com.n3twork.dynamap.DynamoRateLimiter#setConsumedCapacity(java.util.List) vs com.n3twork.dynamap.DynamoRateLimiter#setConsumedCapacity(com.amazonaws.services.dynamodbv2.model.ConsumedCapacity)
! at [Source: UNKNOWN; line: -1, column: -1]
! at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:302)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:268)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
! at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
! at com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:581)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:539)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
! at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
! at com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:558)
! at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:188)
! at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:28)
! at com.fasterxml.jackson.databind.DeserializationContext.handlePrimaryContextualization(DeserializationContext.java:765)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:550)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
! at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
! at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:591)
! at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4733)
! at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4307)
! ... 66 common frames omitted
! Causing: java.lang.IllegalArgumentException: Conflicting setter definitions for property "consumedCapacity": com.n3twork.dynamap.DynamoRateLimiter#setConsumedCapacity(java.util.List) vs com.n3twork.dynamap.DynamoRateLimiter#setConsumedCapacity(com.amazonaws.services.dynamodbv2.model.ConsumedCapacity)
! at [Source: UNKNOWN; line: -1, column: -1]
! at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4314)
! at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:4245)
! at com.n3twork.dynamap.DynamapBeanFactory.asDynamapBean(DynamapBeanFactory.java:33)
! at com.n3twork.dynamap.DynamapLoadService.loadItem(DynamapLoadService.java:92)
! at com.n3twork.dynamap.Dynamap.batchGetObject(Dynamap.java:387)```
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.