FEATURE REQUEST: batch mode about kafka-connect-hdfs

[ewencp]

@allxone The connector can't actually support this directly itself. We could effectively have it exit by triggering an error that causes the framework to shut it down. However, it would be marked as failed and the connector itself would continue to exist. To do this correctly would also require framework level support for connectors that have a definitive ending point. But this is also antithetical to Connect's design, so I'm not sure it's something we'd want to support. Is there a reason you wouldn't just use the connector as is and set the file rotation interval to match your desired frequency?

[allxone]

@ewencp thanks for the clarification.
I understand my request could result in an anti-pattern against streaming ingestion standards and this is confirmed also by the fact that I wasn't able to find any tool with the characteristics I'm looking for.

My requirement is reducing the number of moving parts out of the Oozie workflow control: if the HDFS connector is down the workflow would simply miss data without recognizing the issue.
Being able to specify at the connector startup the ending offset after which it would exit gracefully would be perfect for my use case. With that option I could preliminary query Kafka for the latest topic offset and start the connector knowing it will always exit as soon as it finish consuming the latest messages.

BR
Stefano

[ewencp]

@allxone Yeah, usually where we've seen this request is someone trying to map an existing batch workflow to Connect.

We don't expose the right metrics for this currently (at least not via Connect, you could get them from the internal producer and consumer we use), but the other way to accomplish this would be to start the connector and then just wait for it to hit a steady state of processing 0 msg/s. I realize that might seem like a bit of jumping through hoops compared to just having the connector stop, but it would work for any connector and doesn't require any support from connector or framework (beyond stuff that we want to support anyway).

Hi, could you give a little more insight, or some direction to some documentation, on how I might do this:

"...but the other way to accomplish this would be to start the connector and then just wait for it to hit a steady state of processing 0 msg/s."

[levzem]

closing as this has been provided a workaround