The last few records in tempFiles can't flush to HDFS when stream in kafka in paused about kafka-connect-hdfs

[xianzhen]

We are encountering the same issue.

[Ishiihara]

That is the current design. I think the workaround here is to set flush.interval.ms to a reasonable value? Also, after #87 is merged, the scheduled flush can also be used as a workaround.

[skyahead]

We faced the same issue, and we tried setting flush.interval.ms but it did not help if the stream is paused. The reason is shouldRotate() is only checked inside the while(!buffer.isEmpty()) loop. But when the stream is paused, the buffer is also empty.

In our case, our stream gets data every 5 minute and we flush.size is 50K in size. So, we always have 0 to 50K records kept in temp files until new data is coming.

Another issue we saw is that the first files per topic contain only one record, reason being lastRotate is zero and now - lastRotate >= rotateIntervalMs is always (or most likely) true.

I sent a PR to fix these issues.

Please have a look if it makes sense.

[wjptak]

We have the same issue, using the latest Confluent Kafka. Thanks for the PR!!

[cotedm]

This was resolved by #100