File not written when partition changes about kafka-connect-hdfs

[Ishiihara]

That is the current behavior. We currently closes all files under a topic partition once the flush size is reached. In case of time based partitioner, when a new file is created, the old file can be safely closed as no more data will be written to the old file. Plus, we know that at any given point of time, we only have one file open and I think we should allow triggering a file commit once that file is closed.

One work around is to set the flush.interval.ms to be roughly the same value as your partition size. This ensures that the file is closed when the flush interval is reached.

[13h3r]

@Ishiihara, do you agree that it will be good to close file when partition changed? I can provide PR

[13h3r]

ping @Ishiihara

[levzem]

closing as this is addressed by discussion in #80 and by PR #87