gyfora / streamkv Goto Github PK

Updating currently is not supported (consecutive get and puts do not achieve this). An update is in principle a 'put' operation with a reducer. If a key is non existent it puts the value (equivalent to reduce with the identity value), otherwise it replaces the current value with the reduced one.

Currently the KVStore API is tied to the Flink Java types (Tuple2s mostly) which makes it very inconvenient to use in Scala programs.

A Scala API should be provided that will operate on proper Scala tuple types:

• Add scala KVStore
• Implement mappers to convert from scala tuples to KVOperations
• Implement custom serializers for the scala tuples (if necessary!) to handle null values

The MultiGetMerge function is currently untested. Proper merging and state disposal should also be tested for this operator.

There are currently no tests for checking the checkpointing and recovery mechanism for the KVStores. This should be implemented similarly to the Flink checkpointing tests:

https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/checkpointing/StreamCheckpointingITCase.java