Incorrect handling of exceptions during checkpointing. about amazon-kinesis-connectors

[akumariiit]

@knorwood i am on below version

com.amazonaws
amazon-kinesis-client
1.9.0

and i am still getting the same exception

com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException: Can't update checkpoint - instance doesn't hold the lease for this shard

[BobbyJohansen]

I am on 2.1.3 and still seeing this issue.

[msdeploy]

I also get that error. An issue is that my data get inserted to the redshift database even when checkpointer.checkpoint() fails, so the same data are processed again and again (each time inserted to the database since there is no transaction).

[ajanata]

We're seeing this on at least one of our instances every day. Really no update?

[knorwood]

The root cause of this issue is that sometimes a KCL Worker loses a lease and then reacquires a new lease on a particular shard before discarding the old lease for that shard, it then continues to hold the old lease which still holds the old concurrency token. The mismatched concurrency token prevents checkpointing but the Worker never discards the lease because the lease cleanup logic only compares the shard-id.

A fix for this will involve correcting the logic around removing lost leases to also take into account the concurrency token in addition to the shard-id when comparing leases. In the meantime, one way to alleviate the impact of this issue is to increase the failoverTimeMillis which is set to 1000ms by default in the connector library. Increasing the failoverTimeMillis will reduce the chance of a Worker losing and then immediately reacquiring its own lease.

[knorwood]

A fix for this issue was included in the 1.1.1 release. Details of the fix can be seen in this diff