writeRecord wrongly considers a newer record as stale about kafka-connect-hdfs

[Ishiihara]

@vikeshkhanna I think you are talking about that Kafka Connect is not running until a week later and some messages are already removed according to the retention policy. That certainly makes sense as now the expected offset is not correct anymore. One solution will be seek to the beginning of the topics and if that is larger than the latest offset of the data in HDFS and set the offset to that. This will make sure that the offset honors the retention policy.

[vikeshkhanna]

Yes, that is correct. Yeah, the ideal solution would be to update the expected offset in case some data has been dropped by Kafka based on its retention policy. I don't think this is implemented yet, that's why the bug.

[savulchik]

Seems like related to #132

[kkonstantine]

I believe this is not an issue anymore for the same reasons that were given for #132

writeRecord has been simplified in TopicPartitionerWriter and no records are dropped based on offset now.

I'll go ahead and close this issue marked as a bug as well, but again, feel free to reopen and provided any related logs if this is still an issue in the most recent versions.