[ENHANCEMENT REQUEST] Connector doesn't support deleting records from the table about emr-dynamodb-connector

[Tang8330]

+1

[Tang8330]

@itsgrimetime I tried to modify source to be something like this: https://github.com/Tang8330/emr-dynamodb-connector/blob/dynamo4.2/emr-dynamodb-hadoop/src/main/java/org/apache/hadoop/dynamodb/DynamoDBClient.java#L151

However, it's not quite working and is still inserting, despite me playing around with the default state.

Any ideas? I'm happy to make the necessary changes.

[Tang8330]

my function looks something like this:

JobConf jobConf = new JobConf(sc.hadoopConfiguration());
jobConf.set(DynamoDBConstants.INPUT_TABLE_NAME, INPUT_TABLE_NAME);
jobConf.set(DynamoDBConstants.OUTPUT_TABLE_NAME, OUTPUT_TABLE_NAME);
jobConf.set(DynamoDBConstants.DELETION_MODE, "true");

// Should be deleting.
filtered_results.saveAsHadoopDataset(jobConf);

[ajay25]

@Tang8330 Did you ensure that writeBatch()- https://github.com/awslabs/emr-dynamodb-connector/blob/master/emr-dynamodb-hadoop/src/main/java/org/apache/hadoop/dynamodb/DynamoDBClient.java#L202 was indeed invoked?

1. I would suggest adding necessary logging in putBatch to understand what is going on?
2. Also, examine batchwriteresult. Specifically check for itemcollectionmetrics https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html#API_BatchWriteItem_ResponseSyntax to see how many records were affected.

[Tang8330]

I would suggest adding necessary logging in putBatch to understand what is going on?

Adding it in now, I'll test it out now.

examine batchwriteresult

I have difficulties examining the output since the function results.saveAsHadoopDataset(jobConf); is void

[taklwu]

merge as #96

[tuppimax]

Hi Is there any documentation on how to use this with HIVE ?

[satya-thirumani]

Hi,

Is there any documentation on how to use this on EMR with HIVE ?