Data in unfinished files is lost during shutdown about kafka-connect-hdfs

[ewencp]

@13h3r The HDFS connector manages offsets itself, so I suspect you're interpreting the last part of step 2 incorrectly. You are right that offsets are committed for the consumer because the framework does this and we haven't yet exposed a way for the connector to disable it. We have a JIRA KAFKA-3462 for that and it even has a patch, but I wasn't able to review it and merge it before the 0.10.0.0 deadline.

When a task starts, it figures out what data has been delivered to HDFS (by looking at files in the directory and the WAL). Ultimately this is what tells it where to start. You can see that if it is in an initial state, it starts by trying to run a recovery process. As part of that process, it will reset the current offset to the offset where data was actually delivered.

The key to all this is that in step 1, the data is only being written to a temp file. The WAL is used to indicate the temp file should be delivered into its final location. Even if we crash between the WAL entry being written but before it is moved to its final location, the recovery process can handle completing that process.

In your scenario, the temporary data is simply discarded, but we have not actually lost any data. The next task that resumes processing that topic partition will correctly pick up where the last one left off.

[ewencp]

By the way, I'd love to get a full design section into our documentation -- we'd just need to find enough time to get that written up.

[13h3r]

@ewencp thanks for the detailed explanation.

But when we have no files we are unable to reset offset and messages that has been read before are lost. This can be solved by disabling offset committing to kafka, but what about providing API to commit offset from connector instead of disabling it? This maybe useful to monitor delivery lag.

[Ishiihara]

@13h3r We have a PR in Kafka that allows disabling offset commits in Kafka apache/kafka#1139

[Ishiihara]

Close this because of not an issue.