Update to Kafka 0.9 about clj-kafka

[wgb]

Have you started any work on this yet or do you have a branch I could start contributing to? Working on a project using the new security features and would rather just keep using this lib instead of trying to roll another one.

[pingles]

I haven't started working on it- we're still using 0.8 (and probably will be for a while) so I'm v happy for someone else to start it. I've just created a kafka-0.9 branch- please feel free to pull from that :)

[wgb]

Excellent, thanks. It will definitely be a WIP for a while. There were some non-trivial updates/changes in this release.

[pingles]

No worries- I've bumped the version in project.clj to 0.4-SNAPSHOT, I've
also made a note at the top of the README for people to make note of :)

On Mon, Nov 30, 2015 at 2:13 PM, Worth Becker [email protected]
wrote:

Excellent, thanks. It will definitely be a WIP for a while. There were
some non-trivial updates/changes in this release.

—
Reply to this email directly or view it on GitHub
#72 (comment).

[shmish111]

I've been writing a client lib based on http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client
Basically I wanted something with the common patterns, kind of like curator for ZK I guess. You can still access the KafkaClient in other ways, all the basic Clojure wrappers are there but I just think what that link above shows is what most people will want to do so I wanted to make that easy.
I'm not sure if that would fit into clj-kafka but if you think it does then I'd be happy to convert it and make a PR?

[pingles]

Hi,

So @ragnard has started looking into this (he's using one of our more up-to-date clusters). He's going to start pushing his work to a branch- I think we're going to favour something which does as little extra on-top of the regular Kafka API as possible (albeit exposing sequences etc. so it fits into the rest of the Clojure ecosystem). I'm guessing adding offset committal would probably fit within that scope so its probably worth including in the main lib.

I've no idea how far @ragnard has got yet- if you have a fork it'd be cool to see what you've been up to.

[shmish111]

I started from scratch TBH as I was just doing this in some spare time I had off. I will go ahead and publish it to github when I can and if in the future you decide it would fit well or if I can layer it on top of clj-kafka then I'll do that.

[pingles]

Sounds like a decent way to do it! Great, thank you.

On Fri, Jan 29, 2016 at 11:03 AM, David Smith [email protected]
wrote:

I started from scratch TBH as I was just doing this in some spare time I
had off. I will go ahead and publish it to github when I can and if in the
future you decide it would fit well or if I can layer it on top of
clj-kafka then I'll do that.

—
Reply to this email directly or view it on GitHub
#72 (comment).

[ragnard]

Hey @shmish111 @pingles ,

I just opened #77 for this change. It's WIP, so far I've basically just ported the "new" producer API from the current clj-kafka version to the latest Java client API, and removed a whole bunch of stuff. I'll outline my thoughts around for a new major release in the PR, so feel free to follow that and participate.

[DayoOliyide]

Hi @ragnard

Seeing as the new 0.9.0.0 consumer is incompatible with 0.8.x brokers, is the plan to have
2 permanent branches to reflect 2 versions of the clj-kafka library for 0.8.x and 0.9.x?
Or is the plan to do a final release for 0.8.x and then switch support to 0.9.x?

I'm using a 0.9 cluster and would like to contribute as much as I can to the 0.9 effort.
I've written a wrapper for the 0.9 consumer (needs reviewing/feedback) and would like to do more (e.g tests & documentation).

I would like to know more about the general plans and people's thoughts before I do anything more.

-- Dayo

[Poogles]

Any movement on this?