Write to a topic on S3 file closing about kafka-connect-storage-cloud

[jklukas]

I would also find this useful, and it matches a pattern that Yelp engineers have written about: https://engineeringblog.yelp.com/2016/10/redshift-connector.html

I'm not aware that Kafka Connect provides any built-in support for this, and it would likely be tricky to implement while maintaining the delivery guarantees of the connector.

Another approach would be to enable S3 events on your bucket and have them sent to an SNS topic. You could then write an application that subscribes to SNS or you could use an SNS source connector to get the event data onto a Kafka topic.

[kkonstantine]

Although the workaround is definitely clear, and using Kafka as the only dependency for a connector is is very often meaningful and a good pattern (no new dependency on an external system is introduced, since Kafka is already a dependency for a connector, source or sink), I don't think this workaround is justified yet, in the context of this S3 connector.

Keeping this ticket open, in case you want to discuss this more, but currently I don't find the proposal very appealing.

[brokenjacobs]

This workaround doesn’t work if you are using a non amazon s3 compatible storage system without SNS like functionality. It would be very handy to have some sort of finalizer hook to build off of.